Find My Cat – Power Apps and AI

This post is part of Algorütm PodcastLow-Code No-Code episode:
Audio / Video

No Code / Low Code platforms are becoming increasingly relevant. And with breakthroughs in AI, they are becoming more powerful than ever.

With that in mind, I wanted to make a small tutorial to help anyone get familiar with it, and to try out what they can do. But I mean truly anyone, especially if you are not a software engineer, especially if you don’t have an IT background, I hope you will be able to finish this tutorial and create your first AI driven application. And after that… who knows… your imagination and persistence are the limit 🙂

Find My Cat

Application we are going to create is called Find My Cat. And as the name implies, it’s sole task will be to scan surroundings and report back if it found The Cat. Estimated time to complete is a few hours.

So, let’s start 🎬

Account Creation

We will need to create accounts. If you already have Power Apps account, you can skip this section. If not, please follow instructions below.

Power Apps Account

We are going to use Power Apps Community Plan. With it, you get for free 😀

  • File Storage: 2GB
  • Database Size: 200MB

Note:
To create an account, you will need an email address that has your domain, or your school’s domain. What I mean is that you can’t use your regular @gmail.com or @outlook.com email to login unfortunately. You will need to have @your_domain.com or an email address from your school @your_university.com to sign in. With this in mind, head over to:

https://powerapps.microsoft.com/en-us/communityplan/

And create an account. Click on Get started free.

After you have created an account,
Make sure you also Create an individual environment:

And switch to your newly created individual environment

AI Builder Account

We are going to start Free Trial for AI Builder. Don’t be scared off by the subscription model, or by potentially steep price. We are going to use AI Builder for learning – because it is an excellent starting point. It is easy to use, integrated directly into Power Apps.
Later, if your projects start to live, you will be able to use Microsoft Cognitive Services that have consumption base pricing – which is excellent approach for any project in the beginning. Click here to see a brief overview and sample of Microsoft Cognitive Services pricing.

But for now, while we are learning, we will use Free Trial for AI Builder. And don’t worry, even when Free Trial expires, you will be able to renew it – for free 🙂

Create AI Model

Let’s start with the most exciting item – let’s create our first AI Model.
Login to Power Apps portal, and click on AI Builder / Build.
If you have set up your account correctly, screen should look like this:

Click on Create a database (click here if you don’t see the button)

Fill in the form, and unmark “Include sample apps and data”

After DB creation is finished, you can start AI Builder Free trial

Training the Model

Click on Object Detection

Name your model and click Create

Select Common objects

Create new object and call it “The Cat”

And now you need Cat Photos…

And… you need at least 15 of them. You can take any household item, your favorite figurine, or a cup. Take 15 photos of it from different angles and with different background. That will be enough for this demo.

Let’s upload. Click on Add images

The more you have, the better model will be. 15 is minimum.

Tagging Images

Now we need to tag all the images. Open first one, draw a rectangle that covers your object and tag it as The Cat.

Continue to tag all of the images. System will help you :). After few tags, the UI will start to draw rectangles itself, and you just need to confirm.

When done, click Done tagging

And we are ready to Train

Relax, have a coffee, and come back later 🙂

When training is done, you can actually test it on new set of images and see how it works

Now click Publish, and we can start using it.

Build an App

Now that you have a model ready, you can build an app around it.
Click Use Model / New app

Welcome to App Studio


App Studio is where you will build an app. It looks a lot like Microsoft Excel – and that is intentional.
(below are not the screenshots you will see, this is just a small intro to App Studio)

It has a Formula bar at the top and similar UI. The main difference compared to Excel is that you don’t have Cell Grid, instead, you will place TextBoxes, Buttons and Images wherever you like.

But overall, TextBoxes (Text Input) look and behave in the same way as Cell in Excel would. Quite neat, this reduces learning time.


Ok, let’s go back to Find My Cat app.
Below is the screenshot that you see now: A Canvas App, with AI Builder / Object Detector component on the screen:

Drag the object, and make it cover whole screen.

Now, you can already try out your app 🙂

Press Detect. And show it an image the model never saw before

And the app works. The Cat can successfully be found.

But what happens if one cat goes out, and three cats come back 🙂 ?

We can see that all 3 The Cats where identified, even the upside down one 🙃

Email Notification

Now that app can identify The Cat, it needs to notify us somehow. Let’s send an email whenever The Cat is found.

To add Email capabilities to our app:
Go to Data, type Outlook, and click Connect

Login with existing Outlook account, or create a new one.

Now let’s send an email:

  1. Select ObjectDetector1
  2. Change the Properties to OnChange
  3. Select the formula field

Formula field works in the same way as in Excel.
Start typing Outlook, and autocomplete will pop up, select 'Outlook.com'.

Now start typing SendEmail and select 'Outlook.com'.SendEmailV2

Write “To:” email address, “Subject:” Find My Cat

For the Body, we want to send how many objects we found. We want to send a Sum. Sum function in Power Apps works similar how Sum works in Excel.

Click on fx formula button, and double click Table / Sum

Start typing ObjectDetector1 and select it from the list

Type . (dot) and select GroupedResults from the list

Type , (comma – comma, not a dot) and select ObjectCount from the list

Close remaining brackets, and the formula should look like this
'Outlook.com'.SendEmailV2("my_email@my_domain.com", "Find My Cat", Sum(ObjectDetector1.GroupedResults,ObjectCount))

Let’s try it again

Email Received! Btw, this was the first time that I wrote a program that sends an email, and that worked from the first attempt 🙂

Let’s save our project.
Click File / Save As (or Ctrl+S)

Update the Model

What if The Cat had a friend – The Dog 🙂 ?
It would be nice if we would update our program to be able to detect it.

Let’s go back to
https://make.powerapps.com/
And AI Builder / Models and click Edit Model

Create new Object The Dog

Tag new images

Start training the model again.
Wait until new version finishes

Click Publish

Now let’s run the app

Without us modifying the App itself, Model was able to recognize The Dog and App was able to send the notification correctly.

Next Steps

This is the end of our demo. The Find My Cat app shows some important concepts. It introduces AI into No Code / Low Code application development, and shows some basic steps of application creation. But it also covers important usecase. You might think that this kind of app doesn’t have application in real world, but it does. With slight modification, the concepts we used here can be used for managing a store.

If camera is pointing to the shelf that contains products, we can change the logic to send us a notification when the model fails to find an item.

This would mean that Grinder should be restocked.

Also, using a different model, you can recognize invoices (digital or paper), and automate payments – or at least, automate filling of the payment data.

That is it for now. If you liked the topic, and would love to explore more, you can take a look at:
Garmin Location Tracking using Power Platform

Hope you enjoyed.

Credits

Thanks to Sonya the Cat and Mira the Dog for being awesome 🙂

Reference