Using Azure Automation to copy Live Transaction DB to Test

Updating the Test system with actual transactional data from the Live system is a common task. This can be accomplished using scripts. However, in a hybrid IT environment you might want to organize, manage and monitor all your on-premises and cloud based scripts at a central place. Azure Automation is the platform to do this.

Prerequisites

  • Dynamics AX Live and Test installation on Windows Server 2012 R2
  • Azure Account (all services can be used for free in basic mode)

 

          Azure Automation

        In Azure Portal create a new instance of Azure Automation. When the instance was created, use the key to display the keys and URL. You’ll need this to connect your on-premises server with Azure Automation.

      Azure Automation

        Operational Insights

        Azure Automation can be instantly used to script your cloud-based datacenter. However, to manage the on-premises environment you have to connect your local systems with the cloud. This is done via Operational Insights.

        1. In Azure Portal create a new instance of Operational Insights
        2. Logon to Operational Insights Portal and start the “Get Started” checklist
        3. Add the Automation Solutions
        4. Connect the Dynamics AX server by downloading and installing the Agent for Windows

        Operational Insights

        At your on-premises server open PowerShell in admin mode and go to the agents installation directory e.g.

        cd "C:\Program Files\Microsoft Monitoring Agent\Agent\AzureAutomation\<version>\HybridRegistration"

        Import the Hybrid Registration module and register the server in Azure Automation.

        Import-Module HybridRegistration.psd1
        Add-HybridRunbookWorker –Name <String> -EndPoint <Url> -Token <String>

        Add-HybridRunbookWorker

        In this example I’ve used the Name parameter value “Dynamics” at the Add-HybridRunbookWorker Cmdlet. This creates a runbook worker group called “Dynamics” with one assigned on-premises server.

        On-Premises configuration

        Create a folder where to backup the Dynamics AX database. In this example I’m using a directory on the local system drive C:\AxTemp (which of course is not best practice). By default the Ops. Insights Agent runs as Local System. Make sure to give the account “NT Authority\System” appropriate rights in your SQL Server installation to access the Live DB and Test DB.

        SQL Server Security

        Runbook

        In Azure Portal, go to Azure Automation > Runbooks > Create a new runbook > Choose runbook type “Powershell”. Add the following code to your runbook.

        echo "Stopping Services"
        Stop-Service -Name AOS60`$01
        Stop-Service -Name AOS60`$02

        echo "Creating Backup from Live"
        Invoke-Sqlcmd -Query "backup database [Live] to  Disk = N’C:\AxTemp\Live.bak‘ with copy_only" -QueryTimeout 0   


        echo "Restoring Backup to Test"
        Invoke-SqlCmd -Query "restore database [Test] from  DISK = N’C:\AxTemp\Live.bak‘ with file = 1,  move N‘R3Demo1‚ TO N’C:\Data\Test.mdf‘,  MOVE N‘R3Demo1_log‚ TO N’C:\Data\Test_log.ldf‘" -QueryTimeout 0   
           
        echo "Cleanup Backup"
        Remove-Item -Path "C:\AxTemp\Live.bak"   
           
        echo "Starting Services"
        Start-Service -Name AOS60`$01
        Start-Service -Name AOS60`$02

        This will stop both AOS instances. Make sure the AOS service name fits your environment. The service name can be found in the services mmc. For example my Live AOS is named AOS60$01. Make sure to use the ` to escape the $ sign in the AOS name.

        Dynamics AOS name for scripting

        The script will then create a backup from database “Live” to C:\AxTemp\Live.bak. If your live DB has another name, change it to fit your name.

        Next the script will restore the backup to database “Test”. In my case the logical name of the database file is called “R3Demo1” and it’s log is called “R3Demo1_log”. Make sure this fits your installation. You can find the names in SQL Server management studio, by checking the database file properties.

        SQL Server database logical name

        Finally the script removes the backup file and restarts both AOS

        Runbook Execution

        In Azure Automation Runbook editor, save the actual runbook code and click publish.

        Runbook execution

        At the Azure Automation main page go to runbooks, select your newly created runbook and press the start button. On the next page select execution using a Hybrid Worker and select your worker group. In my example it’s called Dynamics (see PowerShell screenshot)

        Runbook execution

        This will submit your runbook to the on-premises server and execute it. You can check the execution by monitoring the C:\AxTemp directory where the backup will be placed. Don’t worry about the warnings when the workbook finishes. Starting the AOS’ takes a while and results in the typical message “service is not responding”.

        Runbook result

         

        More

        • Azure Automation supports timing of runbooks, so you can create a batch in the cloud to copy your data on-premises
        • This example uses a single server installation. However, the script can easily be modified to run on different servers
        • Using the Invoke-SqlCmd Cmdlet you can do all the cleanup work like changing the reporting server instance etc.

        Publish Dynamics Ax documents to SharePoint

        A typical requirement is to publish documents like Invoice, Purchase Order, etc. to an DMS system like SharePoint. However, in Dynamics AX 2012 and lower there is no built-in functionality to publish these documents to SharePoint. But this requirement can easily be achieved without coding.

        Prerequisite

        1. Install (at least) SharePoint 2013 Foundation incl. SP1. Foundation 2013 SP1 includes all the filters required to process PDF files and a basic search center. By default the configuration wizard creates a new team website collection.
        2. Make sure the WebClient window service is installed and running. If not, you may have to install the User Interface and Desktop Experience Feature on your server.
          WebClient Windows Service

        Document Library and Content Type

        In SharePoint  open the Settings image and open the Site Settings. From the User and Permissions group, open “People and Groups”. Select the Group “Owners”, Click “New” and add the Dynamics AX AOS Service account.

        Provide AOS user access to SharePoint document library

        In SharePoint create a new Document Library. On the team website open Settings image (right, on top) > Add an app > Name the document library “Invoice”.

        Go to the newly created document library. Open the Library ribbon, and edit library with SharePoint Designer. SharePoint Designer is not installed by default, and you may have to download and install it first.

        Open with SharePoint Designer

        After SharePoint Designer was installed, click again on “Edit Library” with SharePoint Designer. Within SharePoint designer, select Content Types and create a new Content Type. Call it “Invoice” and make sure to select the parent content type form “Document Content Types” and parent content type “Document”. Save your modifications.

        New content type for Invoice

        In the navigation, select Lists and Libraries and open the Invoices library. Make sure the option “Allow management of content types” is enabled. At the “Content Types” group at the bottom, open “Add” and select your newly created Invoice content type. This will add the content type to the document library.

        Add Invoice content type to library's managed content types

        Select the Invoice content type at the content types group. At the ribbon select “Set as default”. Now every new document uploaded to the library will be an Invoice type. Save your modifications and go back go SharePoint.

        Set invoice as default content type

        At the Invoices document library in SharePoint, open the Library Ribbon and choose “Open in Explorer”. This will open a new instance of windows explorer, showing the content of the document library.

        Open Library with Windows Explorer

        Click in the address bar and copy the address.

        image

         

        Post and Publish Sales Invoice

        Open Dynamics AX 2012 and go to “Accounts Receivable” > Periodic > Sales Update > Invoice. Change the Quantity to Packing Slip. Make sure the options “Posting”, “Late Selection” and “Post Invoice” are enabled.

        Post Sales Invoice

        Open “Select” and make sure the selection is configured to identify all Delivered sales orders. Click OK and save the selection.

        Select only delivered sales orders

        Open Printer Setup > Invoice. Select File as Target and choose File Format PDF. Open the File Name Dialog and provide the Document Library Address as Target. Repeat this step for the Invoice (Copy).

        Print to SharePoint

        Back at the Posting Invoice Dialog, open Batch processing (at the bottom). Enable batch processing and modify the recurrence to “No End” and repeat all 5 minutes.

        Batch processing

         

        Test

        In Dynamics AX create one or more new sales order and post the packing slip. Wait until the Post Invoice batch job runs. The sales orders should be invoiced. Go to SharePoint and open the Invoices document library. There you should see the uploaded PDFs.

        Dynamics Ax Invoice Document Library

        Open the SharePoint Central Administration and go to Application > Manage service application > Search Service Application.

        SharPoint Search Service

        Select Content Sources and start the crawl manually. This will force SharePoint to index the PDF files in the Invoices document library. By default the crawler indices the documents on SharePoint periodically without manual interaction.

        Manually start crawl

        Open SharePoint Team Website. In the Search Box, type the Name of Customer. SharePoint will present a list of found documents related to the customer.

        Find Dynamics AX documents on SharePoint

         

        (Optional) Confirm and Publish Purchase Orders to SharePoint

        Create a new document library with an purchase order content type as default:

        Go to SharePoint, open the settings image menu and choose to add an app. Select document library and name it Purchases. Navigate to the new document library, and from the Library Ribbon, select “Open with SharePoint Designer”. In SharePoint designer create a new content type, based on a document content type, and name it Purchase. In SharePoint Designer navigate to the libraries, select the newly created Purchases library and make sure the option “Allow Management of Content Types” is enabled. Below, at the content types group add the Purchase content type you’ve created. Finally select the content type, and from the ribbon in SharePoint designer select “Set as Default”.

        In Dynamics AX go to the Procurement and Sourcing module > Settings > Procurement and Sourcing Parameters > and activate Change Management. This option is required to run the purchase order approval workflow.

        Activate change management

        Next go to Procurement and Sourcing > Settings > Procurement and Sourcing Workflows. Create a new workflow based on the Purchase Order Workflow template. In the Workflow Configuration, add the Approve Purchase Order task to the workflow, between start and end.

        Configure Purchase Order Workflow

        Provide meaningful texts for the Workflow Submission instruction. Double click on the Approve Purchase Order taks to open the details. There assign Step 1 to a user or a user group. For testing purpose, you may assign the approval to yourself. Save and activate the workflow configuration.

        Configure Purchase Order Workflow

        Create a new purchase order. The purchase order starts in the approval state Draft. Submit it to the workflow for approval. This will change the state to “In Review”. Go back to the Purchase Order List Page and refresh the data. It may take a few minutes, depending on your workflow batch jobs configuration, until the purchase order is assigned to you for approval. When the purchase order is assigned for approval, approve the purchase and wait a few minutes until the purchase order has reached the state approved.

        Purchase Order Approval State

        Go to Procurement and Sourcing > Periodic > Purchase orders > Confirmation. Make Sure the Options Posting, Late Selection and Print purchase orders are active. From the Select button on the right, make sure only Approved and Open purchase orders will be selected.

        Confirm Purchase Order

        Open the Printer Setup > Purchase Order > Choose print to file as PDF and provide the path to the Purchases document library on your SharePoint. Repeat this for Printer Setup > Purchase Order (copy).

        Print to SharePoint

        Finally, open the Batch dialog  using the button on the lower right. Make sure to activate batch processing and set the recurrence to no end and recurring pattern to 10 minutes.

        Batch processing

        After a few minutes the purchase order state has change from Approved to Confirmed, and you should see the generated PDFs on SharePoint. These documents will be included within the next search crawl and can be found using the search box at your SharePoint website.

        Purchase order confirmation

        (Optional) Enhance Search Experience

        Open SharePoint Central Administration and click the group Application Management. Go to “Create Site Collection” . Make sure your are using the same Web Application as the Team Website. Call the new site collection “Search” and provide an URL including like http://YOUR_SERVER/site/search. Make sure to select the “Basic Search Center” template from the Enterprise template group. Click OK, this will create a new basic search center.

        Create Basic Search Center

        Open the URL you have provided in a web browser. You should see an almost empty web page with a search box.

        SharePoint Basic Search Center

        In SharePoint Central Administration go to “Application Management” > “Manage Service Applications” > “Search Service Application”. Provide the Search Center URL at the property “Global Search Center URL”. This will instruct SharePoint to redirect all search to the Search Center.

        Global Search Center URL

        Open the Search Center URL in a browser and start a search e.g. for a Customer. This will open the result page. On the top right open the settings  image and click on “Edit page”. This will switch the page to edit mode. On the left side, click the drop down of the refiner web part and click “Edit Web Part”. This will open the Edit Dialog on the right.

        Customize Search Center Refiner

        Click on “Choose Refiners” and make sure SPContentType, File Type and Last Modified Time are selected. Remove all the other refiners from the select. Click OK in the dialog, Apply in the Web Part Configuration Dialog on the right and Stop Edition in the ribbon on the top left.

        SharePoint Search Refiner

        When you use the search center the next time, the result page will allow you to refine by Content Type like Invoice, Purchase Order, etc., File Type like PDF, Word, etc., and Date.

        Dynamics AX Document Search

        SQL Server Transaction Log Shipping does not restore .TRN files

        Recently a customer was concerned to implement a fail over strategy for the Dynamics AX database. SQL Server 2008 R2 Standard Edition was in place. The underlying hardware was not capable to handle database mirroring. Therefore, log shipping and manual failover was chosen. However, the job on the second server refused to restore the .trn files. We managed to reproduced the problem in our lab environment with two SQL Instances (SQL1, SQL2) and a Northwind Demo Database.

         

        Log Shipping Configuration

        There are 2 SQL Servers in place, called SQL1 (primary) hosting a Northwind Database and SQL2 (traget). There are two file shares on the secondary server. LSBackup is used to backup .trn files from the primary server. LSCopy is used by the secondary server to copy .trn files and restore these files to the replicated database.

        image

        The backup at the primary server SQL1 was scheduled to run every 5 minutes. Files older than 1 hour in the LSBackup folder will be deleted.

        image

        The target database at the secondary server SQL2 was initialized by the configuration wizard

        image

        The copy job at the secondary server SQL2 was scheduled to run every 5 minutes. This job copies .trn files from the LSBackup folder into the LSCopy folder. Files older than 1 hour in the LSCopy folder will be deleted.

        image

        The restore job at the secondary server SQL2 was scheduled to run every 5 minutes.

        image

         

        Log Shipping Behavior

        The log shipping configuration seemed to be correct. The backup job on the primary server SQL1 placed the backup and .trn log backup files in the LSBackup folder. The copy job on the secondary server copied the .trn files to the LSCopy folder.

        image

        An initial check of the restore job history showed no errors.

        image

         

        The Problem

        A detailed look at the job revealed that the .trn files were not restored. The backup job reported “Skipped log backup file” because it “Could not find a log backup file that could be applied to the secondary database”"

        image

         

        The Reason

        A maintenance plan was in place on the primary server SQL1. This maintenance plan took a full backup at midnight and a transaction log backup every 15 minutes. The interleaving transaction log backups resulted in holes within the transaction log chain. Therefore the restore job on SQL2 was not able to apply the copied transaction log files.

        image 

         

        The Solution

        At the Northwind DB properties on primary server SQL1, the transaction log replication setting to SQL2 was removed. The wizard also removed the copy and restore jobs at SQL2. The replicated Northwind DB at SQL2 was removed manually. The transaction log shipping configuration was deactivated at the Northwind DB on SQL1.  All files in the LSBackup and LSRestore folders were deleted. The maintenance plan was deactivated.

        Next the transaction log shipping configuration was redone, as described above. Again the target database at SQL2 was initialized by a full backup of the Northwind DB from SQL1. Finally the secondary server SQL2 was able to restore the .trn files.

        image

        Data Upgrade Problem: Failed to create session

        I was recently upgrading an application to Dynamics AX 2012 R3 Cumulative Update 8. However, the data upgrade scripts didn’t load properly. AX presented an error “Failed to create session” and there were no batch jobs scheduled.

        Dynamics AX Failed to Create Session

        There ware 3 partitions in this installation

            • initial Initial Partition
        • ps Public Sector
        • ext Extended
          Three Partitions in Dynamics AX 2012 R3

        A detailed look in the dbo.UserInfo table revealed that the User Admin was only linked to the initial partition. However, the user requires access to all three partitions in order to create update scripts for these partitions.

        dbo.UserInfo User Admin is missing Partition

        The solution was to manually add 2 additional rows to the dbo.UserInfo table for the User Admin and link each row to one of the missing partitions. You may alter 2 existing users in order to get a valid RECID, or just increment the last RecID by 1 and delete the records after the upgrade scripts are finished.

        Smoked! Consumer SSD in 24/7 Server environment

        When a customer asks for storage systems, SSDs are a frequent option to boost performance. However, you’ll definitely face the question why enterprise SSDs are so expensive. Especially in an SMB, where you have to discuss each Cent, the question arises if it would be possible to use cheap consumer SSDs instead. Most storage systems use SAS drives while consumer SSDs have SATA ports. Depending on the controller you might not even be able to include a SATA SSD in your storage system.

        However, I had such a situation where a SSD was used as direct attached storage within an IBM x3300 M4 server system. SQL Server was native installed on the machine and an Ocz Agility 3 SSD was used for tempdb only. After 22167 hours of usage (923 days), the first signs of problems showed up. Tempdb files were corrupt and the OS reported I/O errors accessing the files.

        Conclusion

        Using SSD in a storage system or direct attached storage in a server will significantly improve the system performance. However, don’t use consumer SSD in your system. They will probably fail within 2 – 3 years and you don’t get support for 24/7 environment. Enterprise SSDs a more expensive but more durable. Depending on your support contract you get spare parts up to 5 years.

        smoked ssd

        AIF Error: Activity not found & Service Re-Deployment fails

        A customer recently reported a problem with AIF services. The AIF log at System Administration > Periodic > AIF > Exceptions shows failed AIF calls from an external system resulting in an exception “Activity … not found”. However, the service seems to be active and online.

        Default Voodoo here is to deactivate and reactive the service. However, it turned out that the service could not be reactivated. Dynamics AX reported an error that no activities were found. A detailed look revealed that all activities for this service somehow disappeared

        Missing activities in Dynamics AX AIF service

        The solution is to open a development workspace, navigate to the service node in the AOT and (re-) register the service.

        Re-register AIF service

        After re-register the service from the AOT, all missing activities were available again within the service configuration. The service could be activated without any problems.

        Activities in Dynamcis AX AIF service

        Be aware that other services may also be affected by the same problem. Check if your reports are working, and if not check the BI Service. In case the BIService can not be activated, navigate in the AOT to the service node and re-register the SRSFrameworkService and SSASFrameworkService.

        WPA2-AUTO DHCP Problem with Windows 8

        I recently faced a WLAN problem using Windows 8 devices. The WLAN was configured to use WPA with a pre-shared key. The network was working fine with Windows 7 devices. However, devices running Windows 8 only had limited connection with a auto configured IP address 169.254.0.0/16 and did not use the DHCP configured IP range. The problem was solved by changing WPA2-AUTO to WPA2.

        WPA2-AUTO DHCP Problem with Windows 8