Faster dataset annotation and model evaluation with Roboflow – Made with TensorFlow.js

In this interview, we’re joined by Brad, who’s working on a product that empowers the developers to build their own computer vision applications, no matter what their skill set or experience may be. First, for Brad, tell us more about who you are. Hey, Jason, thanks for having me. If I was to describe myself in one word, it’s a hacker. I’ve basically just been building stuff and putting it online for as long as I can remember. I got my start really in the social gaming sphere. When I was in college, I built a Facebook application, that ended up getting over 10 million users, and I dropped out of school and ran that for about a decade. That really brought me into the augmented reality and computer vision world, where I started working on board game helper apps. I made a AR Sudoku solver and an AR Boggle solver, and launched those and hoped that they’d inspire other developers to create similar things. In talking to some of my friends, what I learned was they didn’t need inspiration, they really needed tooling. That led to a pivot to working on developer tools to help other people create computer vision power apps. Wow, that is quite a story. I love it. That’s very, very interesting. Now, what exactly is Roboflove in? And how does that work with WebML and TensorFlow.js? Yeah, so Roboflove is a developer tool. Basically, we’re trying to help developers give their software the sense of site. So taking image data and turning that into actual information that you can utilize for cool things. So kind of the key insight of Roboflove is that the way that humans gather information from the world around us is primarily through our eyes. That’s not really the case for computers yet. So we’re trying to change that and put this into the hands of all developers, not just companies like Google that can spend billions on self-driving cars, but all the fun use cases that you can do to just add vision to your applications as well. So what that means in practices, we build an end-to-end computer vision tool chain. It’s everything that you need to do everything from collecting images, to annotating them, to creating data sets, to training a model, to then going out and deploying it into the wild and building an app. And we’ve had over 100,000 people that have built on Roboflove so far, including giant companies. In fact, over 2 thirds of the Fortune 100 have used Roboflove for something so far, but also thousands and thousands of individual developers and students that are just building cool hackathon projects in their basements. And in terms of how we use TensorFlow.js, there’s kind of three ways, kind of percolates and spiderwebs it’s the way through our entire product. So first, we use it as a model-assisted labeling tool. So once users have trained a model, they can then use that as a jumpstart for improving that model. So they get predictions back from their model. And when they’re annotating images, that’s their starting point. So instead of starting from scratch and having to draw all the boxes, really, you’re just correcting your model and teaching it new things. You don’t have to teach your model the things that it already knows. It’s kind of the philosophy behind that. And then secondly, we use TensorFlow.js to let users preview their models directly in their web browsers, which is really magical. You hit train and the model comes back and you can hit go. And you just see it right in your browser running on your webcam. And you can try your model in real time, which I think will get a chance to preview here in a second. And then third, we kind of wrap that functionality in a developer SDK. And we let them build their own applications that they can deploy to their users browsers to create all sorts of really cool things. Wow, that is truly amazing. And 100,000 users, that’s incredible. So you might be our largest user of TensorFlow.js that I’ve interviewed on a show so far. So congratulations on that. Now, I think it’s time to show our viewers how they can potentially use this product. Can we see a demo? So this is an example of how Cardinal Health is using RoboFlow to improve the back rooms of pharmacies. So they have a division in their company that works with pharmacists where one pharmacist can manage several locations remotely over a video stream. And so this runs in the browser on an iPad. And previously, it was just a video chat between the pharmacist and the technician. And then they used RoboFlow to add on a pill counting feature that helps the pharmacists be augmented by the computer vision model. So they don’t have to count from scratch. They can use the model to estimate and then adjust up and down where the model has not gotten the exact count. Wow, that’s really cool. I love how this is solving real world problems in industry as well, which is really great to see. What’s next? Totally. So let’s pivot to another industry use case, but maybe a little bit more fun one. And this is how Wizards of the Coast is using RoboFlow to power a feature inside of one of their games. So when the pandemic hit, people had to switch from playing Magic the Gathering in person to online. And so they acquired this app called Spelltable that assists in playing Twitch remotely. And so they added a computer vision feature to their app with RoboFlow that can identify what the card is that’s being played. And then the players can hover over the top of it and see the full-scale version of that card so that they can read the fine print even if their video stream isn’t high enough resolution. What, that’s wonderful. I’m sure we’ve got some magic for gathering players in our viewership today as well. So that’s awesome then. I love how it’s enhancing the digital experience when you, unfortunately, remote in these days can really bring that to life in a way that’s readable and understandable by all. So that’s really good to see. Awesome. What’s up next? So now we’re going to jump to something even more fun, which is how the viewers and the audience can utilize this right out of the box. So this is a website called RoboFlow universe that we built. As a public space where our users can share their trained models with other users and then those users can build things with those. So this is an example of a playing card model that one of our users created. And so you can see that on the RoboFlow universe website there’s over 100,000 data sets and 10,000 pre-trained models that you can view. This is one of those. And then we give you the code for you to implement this directly into your application. So this model finds playing cards. And then if we can flip over to the movie magic, I actually did a YouTube livestream of taking this and building an application on top of this. So I took the output of the model, which told me what the playing cards were. And I created a Blackjack strategy app that would find Blackjack hands and tell you whether you should hit stay, double down, or split. Wow. So this is basically called counting essentially. Yeah, it’s a very, very unique topic to see. So I mean, yeah, if anyone’s watching this, combining it with some AR glasses could be very interesting. So very cool, very cool. Okay, what’s the next demo? I think you want one more for us? Yeah. So now let’s do a quick demo of Label Assist, which I described earlier as how you use your model’s output to improve the labeling process and make it so that you can iterate more quickly. So here we trained a model to identify cars and then we are adding more images to our data set. So we just invoke the model and turn it on. And then in our browser TensorFlow.js is giving us a starting point by which to label. And so then you just find where the model made a mistake and you can teach it new things instead of having to teach it the same things that it already knew all over again. Very nice. And so are you able to do the transfer learning in the browser of TensorFlow.js been to like update the model on the fly essentially. And then you’ve been using the new version right there in the browser, right? Is that correct? Yeah, totally. You can use that previous checkpoint for transfer learning. And the training for that actually doesn’t happen directly in the browser. But it’s a human in the loop transfer learning maybe. Wow. So it’s really amazing to see how many different ways you’re able to use RoboFlow for this. And of course, for the labeling side of things, that’s super useful to people who are trying to create new models as well. Like anyone anywhere can then start getting a better model to do the labeling in real time as their labeling, which is really incredible. So great welcome, Vats. So I guess at this point, why did you use TensorFlow.js as a possible solution? Yeah, there’s actually a ton of reasons that we use TensorFlow.js. I think primarily the first thing that gravitated us towards us, was just what a magical user experience it is. Being able to just click a button and see a model working on your webcam feed. And that’s incredibly difficult to do if you can’t run something in a web browser. You can imagine the opposite user experience of like telling users to get a certain graphics card and then install a whole bunch of drivers and like get their system set up. And it’s hours versus seconds. It’s just like night and day difference. And using TensorFlow.js creates that aha moment where they can really utilize and understand our product really quickly. But besides that, there’s a whole bunch of business reasons as well. And one of those is cost. By running these models inside the user’s browser, we don’t have to spin up a whole bunch of GPUs in the cloud. The users are using their own machines computing power, which also means that the imagery never leaves their computers. And so we never are seeing the output of their webcam feed when they’re using these models in the browser that saves on bandwidth, that saves us in cost, it improves privacy. It’s kind of like a win-win all the way around. And then there’s another thing which is until actually relatively recently, using GPU accelerated models was really hard outside of TensorFlow.js. You had to have a very specific brand graphics card. But with TFJS, you can use whatever your user has, whether they have whatever brand of graphics card it is, or even if they only have a CPU. It just kind of works. And so that allows us to fulfill our mission of democratizing computer vision and getting this stuff in front of as many developers as we possibly can. Yeah, but there’s a really good answer. And I’m sure you’re given the scale of your product with hundreds of thousands of users using it, but saving quite a bit of money per month if you were to use server-side graphics cards to deal with those inferences and things like that, right? Yeah, I mean, it wouldn’t even be possible. And the reason that we can offer so much of our product for free is because we’re able to offer that onto the user’s machine. And the more users we have, it isn’t really costing us a ton more money. And so that doubles as a way that we can get this out in front of hundreds of thousands of people instead of just a few hundreds. Yeah. And you also mentioned, of course, privacy as well. And given that the medical applications you showed us earlier on with the counting pills and things like this, there’s obvious reasons for privacy in some of these medical situations, too. So I guess there’s a win-win there for you guys. So that’s really awesome to see. So it seems like it’s a great fit. Now, if folks want to try this for themselves, how can they do that? Yeah. So the first thing to do would be explore a robustly universe. Like I mentioned, there’s over 100,000 data sets and over 10,000 pre-trained models for basically every use case that you can imagine there. And all of those models you can actually try in your web browser. That makes a lot more sense for the ones that are detecting whether you have a face mask on or not or playing cards than it maybe does for some of the manufacturing use cases. But it should give you a great idea and launch point for starting your own projects. And then secondly, there’s a whole bunch of projects on GitHub that go one step beyond that and show you how you can take one of those projects and build something. So we showed earlier the Blackjack example. Another example is an app that I actually built last week that helps you parse your drone video. And I trained a model to find solar panels, and then it plots those GPS locations on the map. And so we actually have dozens and dozens of aerial models on Robophilial Universe already. And so it’s not just solar panels that you can plant. You can also find things like swimming pools or you could use this for search and rescue to find people who have gotten lost in the woods or for monitoring forest fires or any of these use cases. It’s kind of like this generic technology. And there’s an open source repo that you can get started with. That’s awesome. I’m a big fan of drones by itself. So that would be definitely very interesting to me. And I guess one extra question here. Where do you see this all going in the future? I mean, our hope is that computer vision becomes a tool in every developer’s tool chest. We’ve reached 100,000 developers so far, but there are millions and millions of developers out there that haven’t tried computer vision yet. And so we make a core product that is free for you to sign up and create your own custom computer vision models that then you can utilize inside of TensorFlow.js. On the technical side of things, right now we support object detection with TFJS. But we’re working on semantic segmentation and instant segmentation and classification and some of these other problem types that will let you do even more in your users web browsers. So stay tuned for that. Awesome. And just to clarify as well, we’re speaking about many of the data sets you’ve got available and better people to use. Are they all reusable by anyone or are there some of them come up like licensing restrictions or how does that work exactly? Yeah. So our business model is if you’re willing to open source your projects, then we give you the product for free. And so most of our users opt into sharing their projects publicly. And we know that’s not feasible for a lot of businesses that think of their data as like a proprietary asset. And so that’s where our paid product comes in of it’s kind of the GitHub model where, if you need private data, that’s what the paid product is for. And if you’re willing to open source it, then it’s free. And so that’s how there’s hundreds of thousands of data sets out there available for people to start with. Awesome. That’s really wonderful, whack. And of course, for those watching, let us know in the comments below how you might use tools like this to bring your ideas to life in your industry. Thanks Brad again for joining me today. And I look forward to seeing even more TensorFlow.js spoke and raised your platform. Thanks so much for having me, Jason. It was a pleasure. Thank you.

AI video(s) you might be interested in …