Tribute makes it easy for friends and family to create meaningful video montages that are given as gifts on important occasions. We feel like sharing your love and admiration is one of the most meaningful gifts that you can give or receive.
A key part of this process is having a very easy to user video editor. We don't need to re-create Final Cut Pro or even iMovie, what we need is a simple user experience so that people (especially those with low technical literacy) can easily combine these individual Tributes into a cohesive, nicely produced video gift.
We use a third party embeddable video editor from Stupeflix (https://developer.stupeflix.com/tasks/factory.video/). While there are a lot of features we like about it, the more control we have over our core technology and infrastructure, the better experience we'll be able to build for our users.
You should go to the link above at this point and play around with the Stupeflix editor to get an idea of its features and workflow. As you can see, it's pretty easy to add new videos or photos to the timeline, change their order via drag and drop, trim videos, etc. Social importing of videos and photos works nicely as well.
The Stupeflix editor embeds in your site using an iframe. There is an API for you to set various parameters like the theme, video resolution, etc. Stupeflix proxies actions back to a video encoding backend. When you click on Quick Preview, they re-encode a low resolution preview of the video on the fly, applying the various text slides, transitions, theme effects, etc. Currently, Quick Previews can be very laggy to start playing back and you also can't scrub through the video as it is being encoded and streamed.
- Expensive - $3 per HD editing session (so $3 per customer and an additional $3 if we have to re-load the timeline to add videos after the final export)
- Doesn't work on mobile web
- Video Editor
- There is a 20 minute time limit for videos created with Stupeflix. This time limit is too short for most of our Tributes, which recieve anywhere from 30-60 minutes of uploaded video. Note: Three Stupeflix themes (classic, raw, humble) can export up to a 40 min. video.
- Can't play back individual videos (high priority)
- Can't scrub through Quick Preview video, have to watch it linearly all the way through
- Quick Preview takes a long time to start for longer video timelines
- Quick Preview requires Flash
- Quick Preview timing is off - for example, set title slide to display for 15 seconds, only shows up for 2 seconds.
- The audio often gets displaced in quick preview and does not match up with the people who are speaking
- Music
- Track volume doesn’t lower consistently when people are speaking in their videos
- Need more flexibility with adding music, ex. right now you can only have a piece of music through the whole Tribute or on one video. Would be great to have the option to stretch the music over any videos the user wants.
- Customization of title slides and text overlays - they lack design polish
While there are some advantages to Stupeflix architecture of proxying commands to server side rendering methods, I think it may be possible to "emulate" all the various functions of the "video editor" on the client side, using either HTML5 video libraries like Popcorn.js plus CSS filters or using WebGL. For example, popcorn.js let's you define sequences (aka playlists) that have start and end cut times per video, so we can smoothly playback multiple videos that have trims applied to them.
Note: There are some issues with this approach on mobile web with concurrent playback of video playlists (iOS) and WebGL with video textures (iOS and some versions of Android).
I'll discuss each approach in more depth, but if we can accomplish this, the video editor will scale well, since all the work will be done client-side and individual videos can be streamed via cloudfront/s3. We also won't need a computationally expensive render operation until the final video is ready to be exported. At this time, I think we can use a hosted ffmpeg service (pandastream, encoding.com, transloadit) or use a worker service like iron.io workers or AWS Lambda to run our own encoding using MLT Framework or FFMPEG directly.
Javascript Developer with experience in HTML5 video, streaming video, WebGL experience (Remote work allowed) Experience with ffmpeg, node.js, video encoding, video remuxing, etc. is a huge plus. Python / Django experience nice but not necessary.
- A simple timeline where users can re-order videos via drag and drop
- The ability to add or remove videos from the timeline
- We will have a JSON schema that defines the video editor timeline. There will be an array that stores Video objects, which have properties like video_url, duration, start_time, end_time, filters, transtions, and other metadata
- Changes made client side will be persisted to the server via XHR to our backend video editor API. This can be triggered by events or via something like backbone models or ember-data, which persist changes to the server when the local model is updated.
- "Live updates" to the video editor timeline
- We use a service called realtime.co (similar to pusher) that can send notifications to the video editor running on the web page. So for example, when a new video is uploaded to a Tribute, the django API handler parses the video and stores it in our database. It then sends a message "new_video" on channel "". The video editor uses the realtime.co javascript library to subscribe to the same channel (<uuid of the Tribute) when the video editor loads. When we receive a new video message, we add the videos data to the JSON schema and update the UI (i.e. add the video thumbnail to the end of the timeline). This all happens without having to reload the page the video editor is on.
- Upload videos and photos from local computer using the UploadCare widget
- Uploadcare gives us a JS callback when the video or photo file has been uploaded to their service. We update the UI locally and send an XHR to our backend API with the update to our timeline schema.
- A list of the submitted videos (in case you delete one from your timeline and need to add the video again)
- There will be a link at the bottom of the video editor that brings up a list of all the videos ever submitted to this Tribute either in the top "preview" region or as a modal dialog
- Display names of submitters and length of videos (this metadata will be passed to the client side via the JSON schema retrieved from the server when first loading the video editor)
- Works on mobile web (Update: we realize there are issues with WebGL in mobile safari right now, i.e. there's a bug regarding compositing video into textImage2d. While down the road it would be great to use the same code on mobile native (via webview), mobile web and desktop web, we are ok for now targeting native solutions (opengl) on iOS and Android until mobile safari and mobile chrome get all the bugs worked out)
- Transitions - fade, dissolve, reveal
- Stupeflix has: fade to black, fade to white, crossfade, blur, cube, sharp cut
- Look at glsl.io transitions
- Quick preview individual files or the whole movie (can scrub through whole movie)
- The service that handles our video recording, Cameratag, can supply us with multiple encoding resolutions and formats (240p, 360p, 480p, 720p, mp4, webm, flv, RTMP, mp4 stream, etc). For the preview of individual videos (via HTML5 video tag) or for the quick preview function (render the webgl scene), we can probably assume we will use the 240p mp4 files, since we don't need a high resolution preview, just a high resolution final product.
- Trim videos - set start and end points via draggable handles
- Similar to the Video.js range slider or similar to how Stupeflix handles trim & cut currently
- All actions are auto-saved
- gap-less (or close enough) playback of videos in the timeline
- asynchronous rendering of the file video - you can proceed to checkout before final video is done rendering
- Video Encoding requirements
- encode multiple input types to single HD mp4
- Reasonable encode times for final video
- For v0, we'd like to build a prototype of the server side encoding solution that we will build out into production in v1. We'd like to evaluate PhantomJS, Slimer, node-webgl or node-webkit as the headless runtime that we can server side render the scene at high resolution (720p) and composite in our 720p videos. Essentially, we use javascript / three.js to "play back" the scene frame by frame. We call out to ffmpeg via node bindings to get the corresponding video frame and composite it into the scene. We then get the composited/rendered frame from the WebGL context and pipe it to a FFMPEG / MLT encoder process. Thus at the end, we have a high resolution 720p MP4 file. 1. We need to validate that this approach works in v0 so that we can build a server side encode service in v1.
- Social import - facebook, instagram, flicker, dropbox, google+ photos, etc
- We can use the Uploadcare javascript widget for this, it already supports importing files from all the common services listed above
- Text / title slides
- title slides with adjustable duration
- text overlays on video clips
- Filters (ala instagram)
- Proposed solution - Server side encoding infrastructure
- User clicks checkout button next to the video editor
- a POST is sent to the /checkout endpoint on our django instance
- the django POST handler sends another POST with the JSON schema for the video (as well as a shared secret) to our AWS "render" server 1. We can either just leave an AWS instance running, or we could look at dynamically spinning up an instance using the new AWS Docker service
- The AWS "server" uses either node.js or python. We receive the JSON schema, check the shared secret and then download the video assets from S3. We then launch a PhantomJS instance, etc. and render frames out to a ffmpeg process. When the encode is done, we save the MP4 to S3 (using s3fs or an S3 API library) and then issue a webhook back to Django on Heroku with JSON that describes the URL of the video, metadata, etc.
- Music
- add creative commons licensed music
- possibly add your own music (depending on copyright issues)
- Music volume lowers when users speaking (almost mutes)
- Can strech music across whole video, or specific videos
- Themes - Themes will probably just be filters, but we could look at implementing themes like what Stupeflix has (they composite videos in OpenGL scenes like a map background or a pin board, used during intros and transitions)
- Rotate video
- adjust volume per video
- auto-level audio across all videos
- auto-edit type features similar to magisto, animoto, etc
- Name of user under submitted video in the timeline
- Easily re-load a previous video editor timeline for users to add another video, even after exporting a final video
I'll write a seperate Gist that evaluates the various approaches (Popcorn + css, webgl, etc) line by line with the requirements listed above.
Popcorn.js provides much of the functionality that we would need to emulate (from the client side) the Quick Preview function of the Stupeflix editor. The sequence functionality handles contiguous playback and trim support. Trim selection could be handled by something like this: https://github.com/danielcebrian/rangeslider-videojs.
We should be able to use Popcorn.js events to trigger fades between videos to emulate transitions (cross fade, dissolve, etc).
Popcorn.js examples and links
- sequence - http://jsfiddle.net/jmTgD/1/
- embed images - http://popcornjs.org/popcorn-docs/plugins/#Image
- embed audio (soundcloud, youtube) - http://popcornjs.org/popcorn-docs/media-wrappers/#soundcloud
- CSS "instagram" filters - http://designpieces.com/2014/09/instagram-filters-css3-effects/
- Gapless playback https://github.com/jameshadley/LifemirrorPlayer
- Linear playlist playback on iOS and Android. On iOS currently, you have to hit play for each video. There may be a hack for this: https://developer.apple.com/library/iad/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/ControllingMediaWithJavaScript/ControllingMediaWithJavaScript.html
- Can we get CSS filters during our video playback to match filters applied via ffmpeg / MLT?
- Can we smoothly pre-load videos so that we can approximate gapless playback or can we download and cache videos in indexeddb?
Canvas and WebGL are an exciting approach to me and I think it will be possible to more closely emulate the server side encode of the video. We have lots of flexibilty to apply filters, cross fade video, etc. GLSL.io (https://glsl.io/) in particular has a bunch of nice transitions we could implement.
The only downside is that there are still issues on iOS 8 with video textures and older versions of Android have dodgy support.
It could also be possible that our WebGL "definition" could be used on the backend to render the final video as well: gre/transitions.glsl.io#56 http://blog.rectalogic.com/webvfx/
Other links: http://codecanyon.net/item/videogl-3d-html5-video-player-with-filters/full_screen_preview/7404949 https://github.com/povdocs/video-transitions https://github.com/brianchirls/Seriously.js https://github.com/glslio/glsl-transition/ http://codecanyon.net/item/videogl-3d-html5-video-player-with-filters/full_screen_preview/7404949
https://github.com/casatt/html5-videoEditor Note: this project doesn't quite run currently, looking at some forks to see if they have it working again and to evaluate performance. Hard to scale as each user using the editor would need scalable avisynth, ffmpeg, etc
https://github.com/diyaratos/html5-videoEditor/commit/67fe13ed343518b373b6bf1d48519cd855cf56d8 https://github.com/lairoeaes/html5-videoEditor/commits/master
- Can we apply transitions or filters on HLS streams?
-
Hosted
-
Encoding.com 1.
-
Pandastream 1. More targeted at traditional video encoding tasks. Need to investigate how it would handle concatenation and filter related work.
-
Transloadit 1. https://transloadit.com/docs/conversion-robots#video-merge 2. Access to ffmpeg filters, but weird API means you can only concatenate groups of three videos at a time, then concatenate those groups (I think because it was designed for advertising videos, as it is pre-roll, video, post-roll in the API).
-
Host our own
-
User existing celery work queue and setup workers on AWS, Linode, etc
-
Iron.io Workers
-
AWS Lambda (node-ffmpeg bindings)
For the host our own approach, I think we should look at MLT Framework. It's a nice abstraction over pure ffmpeg. It has all the non-linear video editor commands we'd need, plus filters, effects, transitions, etc.
Also worth looking at - Movie.py https://github.com/Zulko/moviepy https://github.com/Zulko http://www.reddit.com/r/moviepy/comments/2f43e3/crossfades/
- Would allow us to have human video editors create cool themes and we could render the videos into them http://news.doddleme.com/equipment/after-effects-harnesses-your-network-as-a-render-farm/ https://helpx.adobe.com/after-effects/using/automated-rendering-network-rendering.html http://www.revostock.com/After-Effects-Templates/?a_aid=68588&gclid=CKaus7rI5cECFcFj7AodaS8A0A http://aescripts.com/ http://videohive.net/item/interstellar-after-effects-template/8198907?WT.oss_phrase=after%20effects&WT.oss_rank=8&WT.z_author=Motionworks&WT.ac=search_list http://store.pixelfilmstudios.com/theme/theme-white-stripes http://themeforest.net/collections/285328-after-effects-showcase
http://www.editshare.com/products/lightworks http://sourceforge.net/projects/lives/?source=directory http://xmodulo.com/good-video-editing-software-linux.html
https://github.com/rectalogic/rendermix
https://glsl.io/ https://www.dyne.org/software/frei0r/ gre/transitions.glsl.io#56 http://blog.rectalogic.com/webvfx/ http://freeframe.sourceforge.net/