No Zero Days November is here again and I’m excited to be doing this again. It’s already been a year and GoZen is on the edge of being able to edit actual video’s!!
What is NZDN?
This is a challenge I set out for myself, to work 30 day’s in a row without having any zero day’s on my free and open source video editor, GoZen. Every day of November I’ll be working on this project with this years goal to be able to edit video’s with GoZen by the end of this month. It is a very do-able goal, but this year might be a bit more busy compared to others. ^^”
Day 1
Last month I let my Ko-fi supporters decide what I should be working on first during this month and the majority has voted for having Hardware Decoding in GoZen, so that’s where I’ll begin. Not one of the easiest challenges, but I have run into performance issues with software decoding which makes this necessary to work.
I started by outlining some of the things I should do inside of the issue I made in GitHub, after which I setup my system and I was off for the first hour. Everything was going kind of good. Not too many issues but there were some playback issues. Video’s would start playing slower and slower but I didn’t know why. Right before this I was working on a system which could take NV12 pixel data and make it into YUV420P, which didn’t really work out the way I wanted to. I’ll explain this a bit later on.
But after checking the memory usage I noticed that there was a memory leak which was causing the footage to become slower and slower, was basically my system giving up hahah. The memory leak was easy to fix by assigning some nullptr’s to avoid crashing and cleaning up the video frame data before loading a new one. Memory issue gone, a couple new issues came.
My time
Every day during this month I want to invest a minimum of 30 minutes, preferably one hour minimum, into working on GoZen. I do have quite a busy schedule this month as my second kid is coming in about a week, I have some freelancing jobs lined up as well as I need to increase my income if I want to continue working on GoZen, and with my wife in the hospital for over a week after giving birth … it will be a challenge to do some work and to spend time with my kid so he doesn’t sit in front of the television all day. But I think I’ll be able to handle everything.
What I can’t promise though is uploading video’s Daily, I will write devlogs of my progress daily on this website and I’ll try to make them on the day itself, but again, no promises on that. Only thing I can promise is that I’ll work on GoZen daily! ^^
Glitching video’s
Before we go on I should mention that video files don’t save their frames in RGB pixels, which is needed to display images on screen, but they get saved in a pixel format called YUV. YUV exists out of three array’s, same as RGB (without the alpha layer), but four pixels would use four numbers in the Y array and would only need one U and one V number from their array to make up the color of those four pixels, probably not the best explanation and rather simple but that’s all you need to know for now.
Hardware decoding was almost implemented, this was thanks to all my earlier tries of trying to make this work, but I couldn’t get the last part to work properly which was displaying the image properly. It took a couple of tries and some fiddling with me trying to handle the whole pixel format structures.
The problem with HW decoding
I mentioned NV12, which is the pixel format in which hardware decoders give back data, which is slightly different compared to YUV which is the format which software decoding returns. NV2 isn’t exactly YUV, but they are basically the same. However, for some reason the GPU (hardware) pixel conversion requires a lot more time to process this image data, which is a problem as now using hardware decoding works slower compared to using software decoding.
Measuring times made me realize that it’s converting to RGB which is the culprit, I tried separating the U and V values into separate array’s which didn’t work as then the converting to 2 separate array’s takes up the most time, so I guess that’s where the software decoder is also failing.
4K footage with software decoding gives us 500 to 600 FPS, enable FFmpeg hardware decoding and we are left with 20 to 29 FPS.
The solution
The solution is simple, pass the data to a shader and let the GPU do all the processing as it is faster then the CPU for these kind of tasks anyway, but that will be for tomorrow.
A big thank you to all my Ko-fi supporters for making NZDN possible, and for supporting me and GoZen! https://ko-fi.com/voylin
