This is a stunning body of work you have produced here, with minds like yours I think that the opensource community could easily compete with the corporate overlords we need to find away to keep you financed and out of their hands and set you free to design until your harts content. Thank you for shearing this with us.
but you actually need new FPGA with PCIe 5.0 along with microwave design knowladge to make new and better device😅 so iphone is definetly one of the most intresting device regard of engineering time amound and it have builtin jtag debugger via type-c!📳 Unfortuanley, we don`t have 48-hours long hardware design review..😢
can you make a small playlist/video , on how to select Camera sensors, and make a module out of them , specially using M-Cortex MCU and FPGA , I know FPGA will work better , but I think making a custom camera module using commonly available MCU like RP2040/ARMCortex-M7 / Xtens LX7 Dualcore/ etc; as this kind of content is not present, atleast as a playlist on yt, you will get a good reach, and students like me will learn a lot. Great Content Sir 💯
*Summary* *General Overview* - *0:02**:* Introduction to the USB 3.0 USBC camera project with CCS mount lens. - *0:16**:* Camera body and lens mount are 3D printed from PLA. - *0:43**:* Camera is made of three PCBs. *Hardware Design and Components* - *1:06**:* Describes the layout of the boards: USB port on top, FPGA in middle, and sensor at front. - *1:13**:* Sensor supports up to four lanes of MIPI interface, tested up to 900 Mbps per lane. - *1:26**:* Boards are interchangeable. - *1:53**:* Mentions different types of sensors (IMX 290, IMX 477) being used. - *2:21**:* FPGA board details, includes a Lattice Crosslink NX, configuration memory, and voltage regulators. *Interfaces and Connectivity* - *3:06**:* Connector details and interface specifications (32-bit 100MHz bus, GPIF). - *3:31**:* Additional control lines described (I2C, reset, camera sync signals). - *3:47**:* USB PCB contains Cypress FX3 USB 3.0 controller. *PCB Details* - *4:04**:* Max IC allows for USB-C connector reversibility. - *4:49**:* Six-layer PCB discussed. - *5:17**:* Second attempt at making the camera, mentions previous version and its limitations. *Open Source and Mechanical Construction* - *6:03**:* Open-sourced designs available on website. - *7:16**:* Mechanical construction, 3D-printed lens mount with metal thread. *Software and Future Plans* - *12:04**:* Software compatibility, supports UVC protocol, customizable frame rates. - *13:11**:* Future plans to discuss schematics, FPGA implementation, and limitations. *Technical Deep Dive into PCB* - *15:00**:* Discusses impedance and length matching for MIPI lines. - *15:19**:* Details the layers of the PCB, mentions that data and clock lines are routed only on the top and bottom layers. - *15:46**:* States that signal layers have a ground reference plane adjacent to them. *Power and Voltage Details* - *16:17**:* Notes that the bottom side of the board has a few decoupling capacitors. - *16:47**:* Explains the use of linear regulators for analog power and a switching regulator for digital power in the second camera sensor. *Camera Sensor and FPGA Configuration* - *17:12**:* Specifies a 1.8 volt oscillator for the camera with a reference clock. - *17:24**:* Mentions I2C lines are also 1.8 volt and must be compatible with the camera's voltage. *Image Processing and Configuration* - *33:15**:* Describes how pixel data is packed in different raw formats, with Raw 12 as an example. - *34:37**:* Debayer filter converts raw pixels to RGB, which then gets converted to YUV color space. *Limitations and Future Plans for Image Processing* - *38:13**:* Lattice engineering sample chips have limitations in port assignment; workaround discussed. - *38:58**:* Current design lacks modules for color correction or other image manipulation. *Software and User Interface* - *41:24**:* PC-side software discussed, including exposure and gain controls. - *45:05**:* 35 FPS full frame performance noted. *Additional Notes and Caveats* - *46:48**:* Mentions the need to dismantle the camera to change sensors. - *48:59**:* Discusses absence of sensor detection in the firmware and the need for identification methods. Disclaimer: This summary was created using the GPT-4 model and serves as a condensed version of the original transcript. The transcript was initially divided into six segments and then summarized using bullet points, each with a starting timestamp. I used this prompt: "Summarize as a bullet list. Keep starting timestamp for each bullet point:". The bullet points were subsequently organized into sections with appropriate titles with this prompt: "Split the following bullet list into sections. Create section titles. Keep timestamps.". The text was manually formatted for TH-cam comment markup.
Really curious to see how you do the AWB and AE (working on our own camera design, not dissimilar to what you are doing, and I'm finding making a robust AWB to be a very interesting challenge)
I have uploaded a 4K video, Video Just Got uploaded few minutes ago. TH-cam it still Processing HD video. Check back in few minutes and you will have HD version
This is awesome! any chance for a simpler design being offered that just plugs directly via USB and acts like a webcam? Kinda like those cheap ELP brand CMOS cameras you see on aliexpress that have a plastic lens c-mount on the pcb. I'm looking to build my own to fit my assembly and can't really use any off the shelf solution.
Unbelievably awesome project, mindblowing that you even release such immense genius work! I wanted to build your board with the IMX477 at 100FPS. Do you perhaps know a rough latency from photon to received over USB in milliseconds? Again, I can not put into words how impressed and thankful I am for your work.
There are few components you have take a look for total latency. 1. Camera sensor. 2. FPGA 3. USB controller Camera sensor due to its architecture should have less than one line latency. FPGA ISP Pipeline only has delay / latency of 4 line , if you are talking about 4K 60 FPS then each line is about 7.71us then 4 line would be around 30 us . So after 30 us of frame start my ISP will start producing valid data on output. USB controller and number of byte latency it should receive frame end or a fix number of byte before it send data to PC so for a 4K it should have max 4 Line worth of time in latency. so for 4k 60 FPS total max latency can be 67.71us.
This is amazing! Where did you get the footprints for the IMX sensor? I want to try this with an IMX252 or IMX273. Although first I'll design for FPC cable instead of USB. Thank you
I want to appreciate you for the effort in the Camera and FPGA interfacing. The work gives me hope that I can handle similar work. Please advise me on how to go about this task; I wish to have a camera with FPGA and DDR3 RAM, the interface should have an Input and Output trigger (End of Frame) signal. I don't mind USB 2.0 or USB 3.0. I want it to work such that an external signal from another board will be used to trigger the sensor to start a frame grabbing. The sensor should be the type with adjustable integration (exposure time).After a complete frame is captured, the output trigger should send a signal to the external circuit. and then the frame data can be read. I don't need high repetitive speed, rather a very high frame grabber speed in the range of 1us or less.
Can you make a video on Multi camera (3 or 4) through mini coaxial sync connected to Jetson or other MPU boards ? The problem with MIPI CSI 1/2 is length of cable and USB is BW limited and power hungry. Coaxial or GigaE are possible solutions which am considering for my project but the cost of Serializer and DeSerializer is beyond budget. Hence can you suggest OR make a video on solution with Multi Cam Sensor connected to small Application cores - or all camera Sensors connected to same App processor ? Thanks and great channel
@@karthikbalu5824 There are Many Suppliers for these camera sensors, You can look on internet. Ask Sony. They will tell you exactly who is selling in your area.
@@circuitvalley Do you offer any practical paid course, that would really help as there many details such as how to create these layers and several other decisions u made in terms of selecting correct FPGA, sensor package etc? Thanks a lot for your extraordinary work !!!
Soldering sensor is not very far from soldering BGA, you can look into any sensor's datasheet or Technical reference manual. Solder process is quite straight forward.
@@circuitvalley Thanks for the answer. I have another question related to this one. When I think about soldering the image sensor, I come to a conclusion that every pin should have the same amount of the solder paste. Every pin should be soldered in the same time interval and temperature. Otherwise the image sensor's normal vector won't align with center of the lens causing distorted image. I think there should be some way to avoid it but I could not find for the industiral cameras. There are some alignment machines for CCMs(Compact Camera Module) but they are not applicable to industrial cameras.
@@emredemir549 If you use solder paste will result pretty much same amount paste on every pad. and soldering using oven will result all pins soldering at same time.
Nice work. I am looking for a low cost camera module that can provide pre-bayer data. Without requiring huge investment of processor boards. Many USB3 to MIPI use cheap encoding. I would be happy with 1 frame per second at 320x240 if it were data just after the ADC before ANY adjustments or corrections. Machine vision is useless with cameras that throw away the raw sensor data by bad methods.
You probably want to use a monochrome sensor then or do you need color for your CV application? The question is, if this could support monochrome or raw RGB in the future to better support CV tasks. What a pity, that UVC doesn't support RGB or RAW modes. It seems poor for anything but a webcam. So there should be an option to support a better (more modern) protocol, maybe for a custom API. What about Linux-support (V4L2 or even libcamera)?
hi nice project. whats an application which benefits from this fgpa camera board - is the use for it to do image processing on a fpga? can you connect it to a raspberry pi? is there a way to make work for a thermal camera too?
@@Gerald-iz7mv He told in the video, that the goal is to have a modular design, which could be enhanced with custom or different available boards for various sensors. Also it should allow you to make use of all available camera features to e.g. get higher FPS with reduced resolutions or to implement a custom white balance, exposure control etc.
I have made multiple attempts to reach this project, First I tried to make UVC stack on PIC32, and then small VGA camera on on PIC32, Then I used lattice macxo3 with IMX219, This was basically 4th attempt and recursive change.
Yes It can be done with any sensors, If you watch video carefully, I already have IMX290 Camera module shown in the video. You can look at IMX290 camera module on the blog post.
Has anyone successfully reproduced this project? I've been struggling with this project for a long time without success. The FPGA is programmed, and the fx3 runs successfully (the PC can recognize the usb device), but the image cannot be obtained using webcamod8.8. I guess there are two reasons: 1. There is a problem with the driver configuration of fx3 and imx219 or imx477. 2. There is a problem with the fpga configuration code. Hope that someone who has successfully copied can provide suggestions or comments, thank you!!!
Why can't the fpga directly handle the usb communication through a level shifter? Is there enough io from the fpga to connect to other interfaces like pcie instead?
First USB is complicated it would need pretty large FPGA for no gain. Second USB 3.0 will need special high speed transreceiver. It would be pretty counter productive, may save some space but will definitely not save money
Hello, this is technically way above my hobbyist amateur head however, you do a great job trying to help me understand extremely complex topics! ** I previously watched some videos related to distributed aperture camera arrays using multiple MIPI CSI Sensors. Is this something you've considered looking into? It would be interesting if multiple Low Res & wide dynamic range (WDR) sensors could be aggregated into a single large sensor with a massive super telephoto (First Surface reflector) telescope lens for long range high-resolution mini CubeSat applications or more of a wide-angle range for persistent stare drone or satellite Surveillance? Is this something you've looked into or considered building? Thanks!
This is a stunning body of work you have produced here, with minds like yours I think that the opensource community could easily compete with the corporate overlords we need to find away to keep you financed and out of their hands and set you free to design until your harts content. Thank you for shearing this with us.
The only real tech TH-camr on the platform. I don't care about the new iphone, FPGA, MIPI, PCIe is where the real tech is
but you actually need new FPGA with PCIe 5.0 along with microwave design knowladge to make new and better device😅 so iphone is definetly one of the most intresting device regard of engineering time amound and it have builtin jtag debugger via type-c!📳
Unfortuanley, we don`t have 48-hours long hardware design review..😢
its crazy how, detailed and high level content is available on youtube now a days, thank you for adding another branch to my knowledge
Glad you enjoy it!
Wow. This is an underrated channel. Such high quality content. Very well done!
How has this channel not gone viral and blown up?
can you make a small playlist/video , on how to select Camera sensors, and make a module out of them , specially using M-Cortex MCU and FPGA , I know FPGA will work better , but I think making a custom camera module using commonly available MCU like RP2040/ARMCortex-M7 / Xtens LX7 Dualcore/ etc; as this kind of content is not present, atleast as a playlist on yt, you will get a good reach, and students like me will learn a lot.
Great Content Sir 💯
Absolutely INCREDIBLE video.
Thank you SO much for such a clear explanation. Much respect to you, young man!
I love this so much! You do such a good job explaining such a crazy complex board! Can't wait to see more!
This project truly is incredible! Brilliant work
Maybe we can crowd-fund manufacturing a little series of this magnificent device? How does this organizes this days?
Really appreciate your effort and your willingness to make the hardwork opensource.....hatsoff mate
*Summary*
*General Overview*
- *0:02**:* Introduction to the USB 3.0 USBC camera project with CCS mount lens.
- *0:16**:* Camera body and lens mount are 3D printed from PLA.
- *0:43**:* Camera is made of three PCBs.
*Hardware Design and Components*
- *1:06**:* Describes the layout of the boards: USB port on top, FPGA in middle, and sensor at front.
- *1:13**:* Sensor supports up to four lanes of MIPI interface, tested up to 900 Mbps per lane.
- *1:26**:* Boards are interchangeable.
- *1:53**:* Mentions different types of sensors (IMX 290, IMX 477) being used.
- *2:21**:* FPGA board details, includes a Lattice Crosslink NX, configuration memory, and voltage regulators.
*Interfaces and Connectivity*
- *3:06**:* Connector details and interface specifications (32-bit 100MHz bus, GPIF).
- *3:31**:* Additional control lines described (I2C, reset, camera sync signals).
- *3:47**:* USB PCB contains Cypress FX3 USB 3.0 controller.
*PCB Details*
- *4:04**:* Max IC allows for USB-C connector reversibility.
- *4:49**:* Six-layer PCB discussed.
- *5:17**:* Second attempt at making the camera, mentions previous version and its limitations.
*Open Source and Mechanical Construction*
- *6:03**:* Open-sourced designs available on website.
- *7:16**:* Mechanical construction, 3D-printed lens mount with metal thread.
*Software and Future Plans*
- *12:04**:* Software compatibility, supports UVC protocol, customizable frame rates.
- *13:11**:* Future plans to discuss schematics, FPGA implementation, and limitations.
*Technical Deep Dive into PCB*
- *15:00**:* Discusses impedance and length matching for MIPI lines.
- *15:19**:* Details the layers of the PCB, mentions that data and clock lines are routed only on the top and bottom layers.
- *15:46**:* States that signal layers have a ground reference plane adjacent to them.
*Power and Voltage Details*
- *16:17**:* Notes that the bottom side of the board has a few decoupling capacitors.
- *16:47**:* Explains the use of linear regulators for analog power and a switching regulator for digital power in the second camera sensor.
*Camera Sensor and FPGA Configuration*
- *17:12**:* Specifies a 1.8 volt oscillator for the camera with a reference clock.
- *17:24**:* Mentions I2C lines are also 1.8 volt and must be compatible with the camera's voltage.
*Image Processing and Configuration*
- *33:15**:* Describes how pixel data is packed in different raw formats, with Raw 12 as an example.
- *34:37**:* Debayer filter converts raw pixels to RGB, which then gets converted to YUV color space.
*Limitations and Future Plans for Image Processing*
- *38:13**:* Lattice engineering sample chips have limitations in port assignment; workaround discussed.
- *38:58**:* Current design lacks modules for color correction or other image manipulation.
*Software and User Interface*
- *41:24**:* PC-side software discussed, including exposure and gain controls.
- *45:05**:* 35 FPS full frame performance noted.
*Additional Notes and Caveats*
- *46:48**:* Mentions the need to dismantle the camera to change sensors.
- *48:59**:* Discusses absence of sensor detection in the firmware and the need for identification methods.
Disclaimer: This summary was created using the GPT-4 model and serves
as a condensed version of the original transcript. The transcript was
initially divided into six segments and then summarized using bullet
points, each with a starting timestamp. I used this prompt: "Summarize
as a bullet list. Keep starting timestamp for each bullet point:". The
bullet points were subsequently organized into sections with
appropriate titles with this prompt: "Split the following bullet list
into sections. Create section titles. Keep timestamps.". The text
was manually formatted for TH-cam comment markup.
Very impressive -- all steps are clearly explained
I am very glad about TH-cam that it is suggesting some good things to me
Really curious to see how you do the AWB and AE (working on our own camera design, not dissimilar to what you are doing, and I'm finding making a robust AWB to be a very interesting challenge)
subscribed in 1m28s, cool stuff!
Woww!! Great work!! Thank you for sharing the work and knowledge!
i wish i could buy this. one of the most incredible videos i've seen
amazing work, thank you for this contribution
Great analysis! (as always!) Great stuff...Thanks !! (ps. poor 360p video quality please upgrade to 1080hd!)
I have uploaded a 4K video, Video Just Got uploaded few minutes ago. TH-cam it still Processing HD video. Check back in few minutes and you will have HD version
This is wildly impressive!
This is awesome! any chance for a simpler design being offered that just plugs directly via USB and acts like a webcam? Kinda like those cheap ELP brand CMOS cameras you see on aliexpress that have a plastic lens c-mount on the pcb. I'm looking to build my own to fit my assembly and can't really use any off the shelf solution.
Unbelievably awesome project, mindblowing that you even release such immense genius work!
I wanted to build your board with the IMX477 at 100FPS. Do you perhaps know a rough latency from photon to received over USB in milliseconds?
Again, I can not put into words how impressed and thankful I am for your work.
There are few components you have take a look for total latency.
1. Camera sensor.
2. FPGA
3. USB controller
Camera sensor due to its architecture should have less than one line latency.
FPGA ISP Pipeline only has delay / latency of 4 line , if you are talking about 4K 60 FPS then each line is about 7.71us then 4 line would be around 30 us . So after 30 us of frame start my ISP will start producing valid data on output.
USB controller and number of byte latency it should receive frame end or a fix number of byte before it send data to PC so for a 4K it should have max 4 Line worth of time in latency.
so for 4k 60 FPS total max latency can be 67.71us.
Very elegant build
This is amazing! Where did you get the footprints for the IMX sensor? I want to try this with an IMX252 or IMX273. Although first I'll design for FPC cable instead of USB. Thank you
Where did you source the technical documents for the image sensors and source the image sensors themselves?
Is it better to use a matte black PCB for the sensor board? Guess reducing stray reflections should be helpful for the image quality.
I want to appreciate you for the effort in the Camera and FPGA interfacing.
The work gives me hope that I can handle similar work.
Please advise me on how to go about this task;
I wish to have a camera with FPGA and DDR3 RAM, the interface should have an Input and Output trigger (End of Frame) signal. I don't mind USB 2.0 or USB 3.0.
I want it to work such that an external signal from another board will be used to trigger the sensor to start a frame grabbing. The sensor should be the type with adjustable integration (exposure time).After a complete frame is captured, the output trigger should send a signal to the external circuit. and then the frame data can be read.
I don't need high repetitive speed, rather a very high frame grabber speed in the range of 1us or less.
Can you make a video on Multi camera (3 or 4) through mini coaxial sync connected to Jetson or other MPU boards ?
The problem with MIPI CSI 1/2 is length of cable and USB is BW limited and power hungry.
Coaxial or GigaE are possible solutions which am considering for my project but the cost of Serializer and DeSerializer is beyond budget. Hence can you suggest OR make a video on solution with Multi Cam Sensor connected to small Application cores - or all camera Sensors connected to same App processor ? Thanks and great channel
wow very nice. I wonder how this camera would work with machine vision? 😎 Thank you.
WOW! DIDNT EXPECT SUCH A AWESOME VIDEO
Glad you liked it!!
@@circuitvalleypls let me know where to buy this camera sensors?
@@karthikbalu5824 There are Many Suppliers for these camera sensors, You can look on internet. Ask Sony. They will tell you exactly who is selling in your area.
@@circuitvalley Do you offer any practical paid course, that would really help as there many details such as how to create these layers and several other decisions u made in terms of selecting correct FPGA, sensor package etc? Thanks a lot for your extraordinary work !!!
nice video, where you did order/manufacture this nice PCBs?
Great work. Just wondering how did you manage to solder image sensor which (I think) is extremely sensitive device?
Soldering sensor is not very far from soldering BGA, you can look into any sensor's datasheet or Technical reference manual. Solder process is quite straight forward.
@@circuitvalley Thanks for the answer. I have another question related to this one. When I think about soldering the image sensor, I come to a conclusion that every pin should have the same amount of the solder paste. Every pin should be soldered in the same time interval and temperature. Otherwise the image sensor's normal vector won't align with center of the lens causing distorted image. I think there should be some way to avoid it but I could not find for the industiral cameras. There are some alignment machines for CCMs(Compact Camera Module) but they are not applicable to industrial cameras.
@@emredemir549 If you use solder paste will result pretty much same amount paste on every pad. and soldering using oven will result all pins soldering at same time.
Nice work. I am looking for a low cost camera module that can provide pre-bayer data. Without requiring huge investment of processor boards. Many USB3 to MIPI use cheap encoding. I would be happy with 1 frame per second at 320x240 if it were data just after the ADC before ANY adjustments or corrections. Machine vision is useless with cameras that throw away the raw sensor data by bad methods.
You probably want to use a monochrome sensor then or do you need color for your CV application?
The question is, if this could support monochrome or raw RGB in the future to better support CV tasks.
What a pity, that UVC doesn't support RGB or RAW modes. It seems poor for anything but a webcam.
So there should be an option to support a better (more modern) protocol, maybe for a custom API.
What about Linux-support (V4L2 or even libcamera)?
Impressive!
Great Job!
hi nice project. whats an application which benefits from this fgpa camera board - is the use for it to do image processing on a fpga? can you connect it to a raspberry pi?
is there a way to make work for a thermal camera too?
Application mostly for Industrial application, Automated tests. You can connect camera to Raspberry PI over USB
@@circuitvalley whats the benefit using it in industrial applications or automated tests? energy efficiency, fps, weight or something else?
@@Gerald-iz7mv He told in the video, that the goal is to have a modular design, which could be enhanced with custom or different available boards for various sensors.
Also it should allow you to make use of all available camera features to e.g. get higher FPS with reduced resolutions or to implement a custom white balance, exposure control etc.
Nice video, keep it up, thank you :)
you can probably sideload test pattern directly from your fpga not from your sensor.
This is really impressive did you draw from another open source project.
I have made multiple attempts to reach this project, First I tried to make UVC stack on PIC32, and then small VGA camera on on PIC32, Then I used lattice macxo3 with IMX219, This was basically 4th attempt and recursive change.
what pcb manufacturer do you use ? the boards look very clean
PCBs are made by JLC
Fantastic!!
good work
Great work!
could this be done with other sensors like the IMX290 or IMX533?
Yes It can be done with any sensors, If you watch video carefully, I already have IMX290 Camera module shown in the video. You can look at IMX290 camera module on the blog post.
Thank you share. Please Turkish subtitle.
This is a very difficult job. You are an expert. I sent you a personal email
Where did you get the camera sensors.. i searched everywhere I cannot get IMX533 mono sensor
Nice job
did the fpga,sensor heating up?i didnt see any heatsink in your design here,when running in 4k sensor heats up and need to be heatsink or not?
ippressive dude wow.
awesome)
Has anyone successfully reproduced this project? I've been struggling with this project for a long time without success.
The FPGA is programmed, and the fx3 runs successfully (the PC can recognize the usb device), but the image cannot be obtained using webcamod8.8.
I guess there are two reasons: 1. There is a problem with the driver configuration of fx3 and imx219 or imx477. 2. There is a problem with the fpga configuration code.
Hope that someone who has successfully copied can provide suggestions or comments, thank you!!!
How much did the fabrication cost?
did you ever figure it out?
Why can't the fpga directly handle the usb communication through a level shifter? Is there enough io from the fpga to connect to other interfaces like pcie instead?
First USB is complicated it would need pretty large FPGA for no gain. Second USB 3.0 will need special high speed transreceiver. It would be pretty counter productive, may save some space but will definitely not save money
Hello, this is technically way above my hobbyist amateur head however, you do a great job trying to help me understand extremely complex topics!
** I previously watched some videos related to distributed aperture camera arrays using multiple MIPI CSI Sensors.
Is this something you've considered looking into? It would be interesting if multiple Low Res & wide dynamic range (WDR) sensors could be aggregated into a single large sensor with a massive super telephoto (First Surface reflector) telescope lens for long range high-resolution mini CubeSat applications or more of a wide-angle range for persistent stare drone or satellite Surveillance?
Is this something you've looked into or considered building?
Thanks!
I do have a Multi Camera project in mind. it would take some time you can check back in few months.
Can it tobe set with global shuter method. (Wich the sensor blink all at once not scanning to produce an image or videos)
Global Shutter is feature of Camera sensor, If you use a Camera sensor that has global shutter then you can use global shutter.
What kind of video format output? Is it RAW?
It can output RAW or YUV
u the mvp
what's it for though?
diy bluetooth speaker
200 fps good lord
Idk how to make IMX 383 work😂
Yes IMX383 Will also work
Very interesting topic sir can you please share your LinkedIn ID
Or please can you create a discord or telegram community channel