Object Detection with PowerApps AI Builder

Power Apps recently got the capability to create some computer vision AI models. One of the IMHO most popular ones is called object detection, which is used to detect (predefined) objects on images. For getting started you may use the free edtion of power apps.

Environment and CDM Entity

AI Builder is bound to a Power Apps environment and the Common Data Model. If you don’t have already created an environment, logon to https://web.powerapps.com and create a new environment.

Create new environment

To use the CDM entities you need a new database. In my case, I created a new one with USD and English as preferred language.

Create new database

It may take a while, and you may need to refersh your browser screen, but the AI Builder (Preview) option will appear on the left menu bar.

Create new Power Apps AI Builder Object Detection model

Next, go to Data > Entites and create a new entity for the type of objects you want to identify. In my case, I’m playing around with Nerf guns, therefore I created a new Nerfgun entity. It requires at least a useful name. Feel free to add more fields.

Select entity from Common Data Model

Provide entity data

In a next step you have to provide information about the different elements that shall be identified. In my cases, which Nerfs guns will be on fotos e.g. Rapidstrike, Slingfire, Cyclonshot, etc.

Entity in Common Data Model

There is an option to edit the entity via Excel. However, in my case the Excel addin is not working, and failing to authenticated 😦 If this happens to you, a workaround is to create a simple Power App and edit the entity via Power App.

Fill CDM entity with data using Power Apps

Create and train AI model

In Power Apps go to AI Builder and create a new model for object detection.

Create new AI model for object detection

Provide a name for the model and select the created entity for detection. From the records in the entity select those records that are relevant for detection.

Select object to detect using AI builder

Next comes the time consuming part, take pictures of your objects in different locations, lighting, quality, day time, etc. upload it to the power apps model. You need at least 15 pictures for each element you want to detect.

Upload images to Power Apps AI Builder

Tag each of the pictures with the corresponding object on the picture.

Tag object on images in Power App AI Builder

After uploading and tagging each picture, train the model. This may take a while and will result in a (not very usefull) quality estimation. Right now I didn’t see recall, precision, AUC, or any other more detailed information. If you are satisfied with the models estimated power, publish it so it can be used in your apps.

Train AI model for object detection

Use AI model in Power Apps

It’s easy to use the trained model in a Power Apps application e.g. on the smart phone. Create a new power app with empty layout. From the menu insert the object detection component.

Create a new PowerApps app with AI Builder

Select your object detection model for the component. Save and publish your app, load it on your phone and test it.

Test Power Apps AI Builder in real life

Show SSRS Report in AX 2012 R3 Mobile Device Portal

This post shows how to provide an SSRS report via Mobile Device Portal in Dynamics Ax 2012 R3. The user will be able to scan an Item ID and the MDP page will show the BOMConsistOf Report for this item. Therefore the report will be rendered as HTML.

First, create a file share on the web server where Dynamics AX can save the HTML File. In my case I’ve created an additional folder in C:\Program Files (x86)\Microsoft Dynamics AX\60\Warehouse Mobile Devices Portal0\ , called it “Report” and set the folder properties in windows explorer to be a file share.

Report folder in Mobile Device Portal

In Dynamics AX Development Workspace, extend the base enums WHSWorkActivity and WHSWorkExecuteMode by adding an additional enumeration value called “ItemInformation”. I’m using the label @SYS113931 for “Item Information”.

Next, create a new class that extends WHSWorkExecuteDisplay

class ERPWHSWorkExecuteDisplayItemInfo extends WHSWorkExecuteDisplay
{
}

The MDP page will include 2 steps, first to scan an ItemId and second to create the report and provide a link. Therefore add 3 methods to the class

// Label and text field for an ItemID
container buildStep0(container _con)
{
xSession    session = new xSession();
container   ret = _con;

    ret += [this.buildControl(#RFLabel,’Lbl_ItemId’,’Scan Item ID’,1,”,
#WHSRFUndefinedDataType,”,0)];

ret += [this.buildControl(#RFText,’ItemId’,”@SYS7407″,1,”,
extendedTypeNum(ItemId),”, 0)];

ret += [this.buildControl(#RFButton,#RFOK,”@SYS5473″,1,”,
#WHSRFUndefinedDataType,”,1)];

    return ret;
}

// Disabled text field with ItemID and Label with URL to HTML report
container buildStep1(container _con)
{
xSession    session = new xSession();
container   ret = _con;

    ret += [this.buildControl(#RFText,’ItemId’,”@SYS7407″,1,
pass.lookup(‘Item’),extendedTypeNum(ItemId),”,0,false)];

ret += [this.buildControl(#RFLabel,’A_REPORT’,
http://localhost:8080/Report/bom.html’,
1,”,#WHSRFUndefinedDataType,”,0)];

ret += [this.buildControl(#RFButton,#RFOK,”@SYS5473″,1,”,
#WHSRFUndefinedDataType,”,1)];

    return ret;
}

// Set ItemID range and save report to File
private void createReport()
{
ItemId itemId = pass.lookup(‘Item’);
Query query;
BOMConsistOfController ctrl = new BOMConsistOfController();

    ctrl.parmReportName(ssrsReportStr(BOMConsistOf,Report));
query = ctrl.parmReportContract().parmQueryContracts().
.lookup(‘BOMConsistOfDP_DynamicParameter’);

SysQuery::findOrCreateRange(query.dataSourceNo(1),
fieldNum(InventTable,ItemId)).value(itemId);

ctrl.parmReportContract().parmPrintSettings()
.printMediumType(SRSPrintMediumType::File);

ctrl.parmReportContract().parmPrintSettings()
.fileFormat(SRSReportFileFormat::HTML4_0);

ctrl.parmReportContract().parmPrintSettings()
.parmFileName(@”\\localhost\Report\bom.html”);

ctrl.parmReportContract().parmPrintSettings().overwriteFile(true);

    ctrl.run();
}

Add this code the displayForm method

public container displayForm(container _con, str _buttonClicked = ”)
{
WHSWorkExecute  workExecute = new WHSWorkExecute();
container       ret = connull();
container       con = _con;
int             hasError = 0;
int             startInfologLine;

    pass = WHSRFPassthrough::create(conPeek(_con, 2));
hasError = this.hasError(_con);

    if (pass.exists(#UserId))
{
userId = pass.lookup(#UserId);
}

    startInfologLine = infologLine() + 1;

    switch (step)
{
        case 0:
            ret = this.buildStep0(ret);
step = 1;
break;

        case 1:
pass.insert(‘Item’, conPeek(conPeek(con, 4 + hasError), #data));
this.createReport();
ret = this.buildStep1(ret);
step = 2;
break;

        case 2:
pass = this.resetPassthrough(ret, false);
ret = this.buildStep0(ret);
step = 0;
break;

        default:
break;

    }

    ret = this.updateModeStepPass(ret,WHSWorkExecuteMode::ERPItemInfo,
step,pass);
ret = this.addCancelButton(ret, 1, true);

    return ret;
}

With this code in place the Mobile Device Portal page will show a simple text with the Reports URL. However, to provide a link, some JavaScript code is require. Start Visual Studio (or any other HTML editor) as Admin and open C:\Program Files (x86)\Microsoft Dynamics AX\60\Warehouse Mobile Devices Portal0\Views\Execute\ DisplayIEOS.aspx file. At the init() method add this code to hide the label and add a link.

function init() {

    “use strict”;

    focusFirstEnabledInput();

    var label = document.getElementById(“A_REPORTLbl”);
if(label != null)
{
label.style.display = ‘none’;
var link = document.createElement(‘A’);
link.setAttribute(‘HREF’,label.outerText);
link.appendChild(document.createTextNode(“Report”));
label.parentNode.appendChild(link);
}
}

Make sure to compile your work in X++ and IL and synchronize the Data Dictionary. Restarting the AOS might also be a good idea at this stage.

Finally, configure the new activity to be available in Mobile Device Portal menu. In Dynamics AX 2012 R3 > Warehouse Management > Setup > Mobile Device > Mobile Device Menu Item > Add a new item with Mode:Indirect and Activity Code:Item Information. In Warehouse Management > Setup > Mobile Device > Mobile Device Menu > select inventory and add the newly create menu item.

Open the Mobile Device Portal page, logon, select Inventory Management and Item Information. Provide a valid ItemId which has BOM lines. If you are using Contoso Demo data, item D0007 from company USMF is an example. Next, the SSRS report is create and placed as HTML file. The mobile device portal page will show the link to the HTML report.

SSRS Report as HTML in Mobile Device Portal

BTW: you may use Visual Studio Addins to debug the application

Related Work

Surface RT Business Use Case

Since Microsoft has reduced the price for Surface and Surface RT I’ve bought a 32GB RT device. I was wondering if it would be useful in a business environment. Here are the specs: NVIDIA Tegra 3 Quad Core CPU, 2 GB RAM, 32GB storage / 16GB free, 1366 x 768 at 10.6", wifi, usb 2.0. Surface is a good tablet, however to make it useable in business you need at least a touch cover that gives you a keyboard and touchpad. I also utilized the usb port for a wireless mouse.

Surface RT

The touchcover is a great idea, an almost perfect solution to equip a tablet with a keyboard. It is thin, light, precise and protects the Screen during Transport. However, I’m missing some keys compared to a full featured Keyboard e.g. the print button to make screenshots. As the touchcover has no regular keys, it is hard to type in a dark environment. Of course the touchcover has no back light and you don’t "feel" the keys you’re aiming for. By design it is a tablet with a Keyboard, and you can’t put it on our keys like a laptop.

Office

Here are the good news. Surface RT has a Windows Desktop as you know it from your Laptop or Desktop. It has taskbar, you can pin shortcuts on the Desktop like documents and Computer. Surface RT comes with Office 2013 Home&Student (Preview), which looks like a regular Office 2013 application on a PC. Right now, as I’m writing this post there is no final Outlook RT.

Office 2013 RT

Update: Office Home&Student must not be used for commercial use. Therefore you need to own a seperate office license, e.g. via Office 365

Network Connectivity

The Surface RT tablet has wifi built in, but no WWAN card. If your on the road without WLAN you will have to use your mobile phone instead. Since almost every modern mobile phone can be used as WLAN access point nowadays, I don’t see it as limitation. The tablet has no LAN port, this might be an issue if you’re at a very old school cable based customers. You can configure windows VPN like you know it from your desktop and laptop devices. Surface RT is running the Remote Desktop Service Client that implements RDP 8 protocol. So you get all the cool features like WAN optimization via UDP (if there is server 2012 on the other side of the line). You also get PowerShell v3, but without ISE or –ShowWindow option.

PPTP VPN and RDP Connection

Applications

The platform limits the native execution of applications on the device. You can’t install a Dynamics AX client or other applications on the machine . However, since there is Remote App technology around for a while you can provide applications running on a RDS server (aka. Terminalserver). The RemoteApp Manager has a simple wizard that guides your through the process. Select an application, provide optional parameters e.g. a .axc config file, export the RemoteApp as RDP file and place it on a windows share. Open the RDP file from the Surface RT, authenticate yourself and voila here is Dynamics AX 2012 on a Surface RT:

 Dynamics AX 2012 Remote App on Surface RT

Business Apps

There are many apps available for windows 8 tablets, and some of them are business relevant. There isn’t hardly any useful Dynamics AX app available at the moment. However, here is summary what I’ve installed on my device: ÖBB Scotty for Austrian national railways, Skyscanner to query cheap flights, Holiday Inn IHG hotel reservation app, connect.8 for Xing, Photoshop Express to pimp my photos, Skype, Technet News and Windows Blog Feed Reader, a Facebook app, WordPress app, IBM Presales Advisor, Network Speed Test and Amazon Kindle App.

IBM Presales Advisor App for Windows 8

Windows Mobile & CE

They say Windows Mobile is dead. I agree, its dead on the mobile phone market. However Mobile and CE legacy OS’ are still relevant to develop mobile applications on barcode reader and RFID reader / writer devices. Just to mention Smiley ..

scanner

Make Motorola/Symbol 9090 Scanner scan

I was recently working on a Motorola 9090 scanner device, loaded with Windows Mobile 6.5, developing a .NET Compact Framework application. Most scanner devices automatically trigger the scanner when pressing the scan button and write the decoded barcode text in any text field or document. This device didn’t and there was no preinstalled scanner tool to configure the devices behavior. However, there is a nice application called DataWegde provided by Motorola to make the scanner scan without using any obscure APIs.

Connect a Windows Mobile Device with Dynamics AX

It’s easy to connect a mobile device like a smartphone with Dynamics AX using web services. This tutorial requires Microsoft Dynamics AX 2009 with AIF Webservices installed and Visual Studio for Smart Device development.

Deploy AIF Service

  1. Go to Basic → Setup → Application Integration Framework → Services
  2. Select Refresh button to update service list
  3. Enable CustCustomerService
  4. Generate the webservice

    Aif Services

    Aif Service

  5. To verify your service is working open the IIS manager
  6. Go to → Sites → Default Web Site  → MicrosoftDynamicsAxAif50  → Content View

    IIS 7 Manager on Windows Server 2008

    IIS 7 Manager on Windows Server 2008

  7. Right Click on CustomerSerivce.svc and Browser
  8. You should see a web page with a link to a WSDL file
  9. Follow the link an you should see an XML document containing the service contract

Implement Mobile Device Project

  1. Open Visual Studio and Create a new Solution
  2. Add a new Smart Device Project
  3. Go to References and add a new service reference
  4. Provide the URL of your Dynamics AX CustomerService
  5. Open Form1.cs file
  6. Add a text field (Name: textBoxAccountNum),
    a button (Name: buttonLookup)
    and a label (Name: labelName) to your form
  7. Double Click on the button and implement the service call:
private void buttonLookup_Click(object sender, EventArgs e)
{
    System.Net.NetworkCredential credentials = new System.Net.NetworkCredential();
    credentials.Domain = "YOUR_DOMAIN_NAME";
    credentials.UserName = "YOUR_USER_NAME";
    credentials.Password = "YOUR_USER_PASSWORD";

    QueryCriteria query = new QueryCriteria();
    query.CriteriaElement = new CriteriaElement[1];
    query.CriteriaElement[0] = new CriteriaElement();
    query.CriteriaElement[0].DataSourceName = "CustTable";
    query.CriteriaElement[0].FieldName = "AccountNum";
    query.CriteriaElement[0].Operator = Operator.Equal;
    query.CriteriaElement[0].Value1 = textBoxAccountNum.Text;

    CustomerService service = new CustomerService();
    service.Credentials = credentials;
    try
    {
        AxdCustomer customer = service.find(query);
        if (customer != null &&
            customer.CustTable != null &&
            customer.CustTable.Length > 0)
        {
            labelName.Text = customer.CustTable[0].Name;
        }
    }
    catch (Exception ex)
    {
        MessageBox.Show("Service Call failed",
                        "Error",
                        MessageBoxButtons.OK,
                        MessageBoxIcon.Hand,
                        MessageBoxDefaultButton.Button1);               
    }           
}

You application should look like that:

Windows Mobile 5 Emulator

Windows Mobile 5 Emulator