Showing posts with label Maps. Show all posts
Showing posts with label Maps. Show all posts
(2019-Jan-28) When you work with maps using ArcGIS visual in Power BI, you always have a feeling that it is a tool within another tool. On a surface level, you have options to set data attributes for geo coordinates, coloring and time controls. However, when you go into the Edit mode of ArcGIS, the possibility to adjust your map visualization is expanded to setting base maps, location types, map themes, symbol styles, and pins as well as infographics and reference layers

Reference layers are the additional shape/geo objects that you can to add in ArcGIS Power BI to enhance your data story with more contextual elements related to your existing maps.
Here is an extract from the official ERSI documentation:
"When you add a reference layer to the map, you're providing context for the data you're already displaying. Reference layers can include demographic data, such as household income, age, or education. They can also include publicly shared feature layers available on ArcGIS Online that provide more information about areas surrounding the locations on your map. For example, if your data layer shows the location of fast-food restaurants, you could add a reference layer that shows the proximity of high schools and universities. Reference layers allow you to dig deeper into your data to provide a greater picture of your business information".

How can add my own custom layer (shape) to ArcGIS and use them further down in my Power BI report? This was a point of my interest and a result of questions from other people! To make my further attempts to explore this very topic I owe to this blog post: written by David Eldersveld where he shares very detailed steps of creating custom reference layers to be further found in Power BI:
  Step 1 – Sign in to ArcGIS Online
  Step 2 – Choose the source file from your computer
  Step 3 – Share your feature layer
  Step 4 – Search your reference layer in Power BI

In my new case, I wanted to test out my own geo shapes that I had already created using QGIS application (blogged about this already: So, can I transform my Giza Pyramids shapes into the ArcGIS reference layers and find them in Power BI?

Before you sign yourself into ArgGIS Online, there a few things you will need to decide on what type of account you can use there.

ArcGIS Public Account:
- ArcGIS Public Account is a personal account with limited usage and capabilities and is meant for non-commercial use only.
- You can still create feature layers and maps and further share them publically
- You shared feature layers will be stored in the public Feature Collection.

ArcGIS Organizational Account:
- As a member of an organization, you will have access to the organization's geospatial content that you can use to create maps. You can also share your work with other members of your organization, participate in groups, and save your work.
- You can create feature hosted layers and maps and further share them publically
- You hosted shared feature layers will be stored in the Feature Services.

And here is a very important thing, the only way for Power BI to see your created feature layers is when they are created in your organizational ArcGIS workspace and shared as a hosted feature layer. Public ArcGIS account access won't provide you with this functionality. 

The trial of ArcGIS Online, which would give you an organizational account for 21 days: Your content would be lost after the 21-day period, however.

So, following David's Eldersveld initial set of steps:
  Step 1 – I've accessed ArcGIS and applied for their organizational account trial.

  Step 2 – I've created a new geo item in my ArcGIS workspace and selected a zip file with my shapefiles of Giza Pyramids that I had previously created:

I provided tile and tags for this new item, and I have also selected a checkbox to make this feature a hosted layer.

  Step 3 – By clicking [Share] button I made it available and searchable to ArcGIS map in Power BI:

  Step 4 – Search your reference layer in Power BI
I've added and publicly shared another pentagon-shaped layer that I manually created in QGIS before. Both feature layers looked this way in my ArcGIS workspace content:

And this a culminating moment for me: I was finally able to locate

And use my publicly available reference layers in Power BI:

It is always a rewarding feeling to experience when a quest to validate something unknown results in a successful outcome. However, it's too sad that sharing custom feature layers in ArcGIS Online using personal access account doesn't allow to publish your shared layers to ERSI hosted feature service repository. And yes, currently this is possible through ArcGIS organizational account access only.

Perhaps this will be improved in the future. In either way, after working both in QGIS and ArcGIS tools, this whole GIS technology is no longer rocket science to me. It's only a matter of time to get more experienced with it! :-)
(2019-Jan-14) Dynamic maps with dynamic visualization usually attract my attention, especially when I look at the satellite images of various cloud formations in the upper levels of the atmosphere. Heatmaps could be another example where animation could play a supportive role to point geo data changes as well as certain areas with high or low density of data activity. Basically, animated maps could hold your attention for a few moments in order to reveal an additional data story to you while you're looking at some static facts.

Possible options to make time animations using available maps in Power BI are:
- Use of time slicers, time sliders, other UI temporal controls that would help to interact with existing maps within your reports;
- Change filter values of your dataset time-related attributes (but very quickly :-), it won't be a legit animation though).
- Or use a time animation option of the ArcGIS Map for Power BI.

This particular feature of the ArcGIS Map is often overlooked. Or sometimes, Power BI developers are aware that this mapping functionality does exist there, however, it is still missed in data visualization storytelling. Let's try to explore and become more familiar with this feature of the ArcGIS.

For this example, I will be using an open data set of the Ottawa OC Transpo transit company:, with bus schedules data (bus routes, stop locations, trips for a particular day). In most cases, major cities present this type of data in the General Transit Feed Specification (GTFS) format.

OC Transpo GTFS data is a combination of 7 flat files:
- agency.txt
- calendar.txt
- calendar_dates.txt
- routes.txt
- stop_times.txt
- stops.txt
- trips.txt

Data Model
Those seven files get extracted and become a data model of my Power BI report:

Map Visualization
By using stop_lat and stop_lan geo-coordinates from the stops table, I plot OC Transpo bus stops on my map of Ottawa (capital city of Canada).

I can add the arrival_time column of the stop_times table to the Time fields of the ArcGIS Map visualization, yes, it exists there! Technically you can use Time or DateTime data type fields of your dataset to see it changes over time. The time slider appears as an animation control overlaid on the map. 

And here is the easy tricky part, when you click Play button, the animation shows features on the map in the current time interval. The time slider divides the time data into 10 intervals. When the animation plays, each interval is shown on the map for two seconds. You can also use Pause, Next and Previous buttons to control your time visualization.

Here is a brief glimpse of how it works through a simple animated gif visualization. Please try this with your Power BI report, it's fun! This animated map is now showing me a density of bus stops within a time period that I selected, or I can choose another scenario as well.

This note comes form the official ESRI web site:
When the time animation is playing, you cannot select features on the map; selection is available only when the animation is paused, and you can only select features visible on the map during that time interval. Click Play, forward, or back buttons, or move the slider handles to view and select features from a different time interval. Previously selected features are cleared when the new time interval appears.
So, now you have a way to see your Power BI maps live and moving! 

However, it still doesn't resemble a similar more smooth experience if you had built the same map visualization using Power Map (3D Map) in Excel. Back in 2016, I created this visualization for the SQL Saturday training event in Ottawa, using only Excel application and a similar open dataset; this tool has a way to create an MP4 video file, which later I uploaded to YouTube. Unfortunately, this visualization is not available in Power BI and I have doubts if it ever will be there.

So at the moment, you're given with options to rely on a manual changing of your existing Power BI report filter values, or let the ArcGIS map to automatically move through the time of your dataset, or possibly create a new data visualization in the Power Map in Excel (the latter has not been updated for a long while though). 

Final decisions are always yours!
(2019-Jan-06) In the past, I used to always rely on shapefiles crafted by others to create my map visualizations in Microsoft Power platform. City regions, country province territories, other shapefiles came from open data repositories and constructed by other people. After writing my last blog post on Using WGS 84 shape maps in Power BI and getting more experience with QGIS, I thought that I could create my own map shapefiles and test them in Power BI too.

Hypothetically, I could try to locate and create a polygon of a pentagon-shaped building, or simply follow a circular shape of the corporate office of the Apple company in California. Or maybe, just by going to the southern hemisphere in order to try and replicate a sea-shell-like-roof of the Syndey Opera House in Australia; or reproduce square formation of the Egyptian pyramids.

I've created shapefiles for a couple of those buildings, however, I will show the simplest one in this blog post and lead you into a journey to Pyramids of Giza in Egypt, which from the airview is a simple structure of three square shapes, very easy to start!

About a year ago, I had already blogged about using QGIS tool to create shape figures and test them visually in SQL Server - Geometry Objects in SQL Server using Latitude/Longitude coordinates. So, I won't get into much of details of leveraging features of the QGIS tool, you can try and use other GIS software applications as well.

1) Open Street Maps (OSM) layer in QGIS
Once I open the application and I can add the OSM layer to my project by going to Web > QuickMapServices > OSM > OSM Standard. Then I visually locate three Giza Pyramids on my map.

2) Shapefile layer 
Then I add a new shapefile layer to create three square polygons and name this layer as Pyramid_shapes_wgs84.

Important thing: please make sure to set your Coordinate Reference Systems (CRS) to EPSG 4326 It will provide Latitude & Longitude of your GIS data. This could be done either on your GIS project level or shapefile layer level. Otherwise, If you don't use WGS 84 (EPSG 4326) coordinate reference system, Power BI shape map projection would be visually skewed. 

3) Shapefile conversion to TopoJSON format
With the help of MapShaper, I convert my set of shapefiles
- Pyramid_shapes_wgs84.shp
- Pyramid_shapes_wgs84.qpj
- Pyramid_shapes_wgs84.prj
- Pyramid_shapes_wgs84.dbf
to the new TopoJSON Pyramid_shapes_wgs84.json file.

4) Testing custom shapefile in Power BI
And then I successfully test my own newly created map shapefile by using Shape Map visualization in Power BI.

a) Custom map shapefiles can be created by regular people like me :-) 
b) Power BI is a good tool to validate those shapefiles along with using additional data metrics based on your own data case scenario.

It is my new happy data adventure!
(2018-Aug-30) While preparing for a recent Power BI Toronto meetup session, I found a very valuable whitepaper on Maps in Power BI written by David Eldersveld from the BlueGranite consulting company, and I had used some of the ideas and examples from that document in my presentation.

There is one idea that I couldn't agree more which could be traced and found in many of my Power BI report developments (I admit) and reports developed by others. We've integrated external data source systems, created key metrics in our data model, identified that some of the data elements could be categorized as geolocations in our datasets, and... then we want to start using maps visualizations right away, but do I really need them?

If my dataset contains a customer location info or list of countries where all the tweet messages came from based on a recent marketing campaign then I'm tempted to visualize them. Let's take an example of the following report with the Canadian population by Province. What would be the best way to analyze the population itself, a table, a map, a bar or a pie chart or combination of all of them?

I could easily sort the table by the Population column or look at another non-map visualization (Donut chart, for example, developed by ZoomCharts company) to realize that Quebec is the second largest province by its population.

or I could just select the least populated provinces and territories to find out that their ratio vs. other populated area is less than 1%.

So I wouldn't suggest using Map visualization in Power BI just because my data can allow this to happen. Use other more effective tools or visual components to present your data; combine them with maps, and most of all create a story with your visualizations that will help your audience to understand it better.

It only takes to learn maps visualizations to understand that sometimes something else could be used instead :-)
(2018-July-29) My initial attempt to explore and test the ArcGIS Map paid version continued after receiving a response from the ESRI support team. Initially, I couldn't see all the additional features of the Plus subscription and I wrote about that in my last blog post - ArcGIS Maps for Power BI: Free vs. Paid version

But then I received a response from the Canadian Tech ESRI support team on my request to look into the issue of non-connectivity to Plus subscription content in Power BI. Basically, it communicated that all things were OK with the map itself and I needed to check further with Microsoft Power BI support team.

Well, I was a bit disappointed with this outcome; in addition to this, I received a few comments from other people about their unsuccessful attempts to resolve the very same issue with ESRI support team.

The next day I received two other messages: one with a confirmation that my ArcGIS Map Plus subscription had been activated and another email from a business analyst from the ESRI team in the US with a request to check if I still had any issues connecting to my account and additional help was offered if I needed. This restored my faith in people, and they willing to help others :-) Plus, after quickly checking my original Power BI report, I was able to connect to the Plus subscription of ArcGIS MAP, yes!

So, this is my story of checking all the new features available in the paid version of the ArcGIS Map in PowerBI.


First, the basesmaps add a contextual background to your map geo points. And with the Plus subscription you have access to 8 new basemaps, identified by a Plus badge on the screen:

New basemaps are:

- Imagery
- Imagery with Labels
- National Geographic
- Oceans
- Terrain with Labels
- Topographics
- USA Topo Labels
- USGS National Map (USA)


And my favorite basemep among those eight ones is the National Geographic, it hard to tell why. Probably, because it combines both technical geocoding details like any standards maps, but at the same time, it bears that graphical look from past historical sea voyages into the future endeavors to explore oceans, and I just like it :-)

Reference Layer

With Plus subscription you have access to Live Atlases, which can enrich you map look with other graphical layers on the top of your existent basemep. The search option allows you to examine available layers that you can use. Currently, most of the layers are US based data, but I found a North America Ecoregions layer that showed Canada in so many different colors. Just take a look!



Infographics are just "floating" data cards that you can add to your map with some statistical data  (population, age distribution, income levels, etc.) to support either your whole map visualization or a part of it.

The beauty of those infographics data cards is that their content is updated depending on what you're doing with your ArcGIS. If you select one or more features on the map, the cards will show demographic information for an area around each selected location. See, how the Toronto infographics data gets changed when I zoom my view in to a core downtown area.


I won't be making any conclusions at the end. If you prefer using MapBox in Power BI, it's your choice; if you want to give a try to ArcGIS Plus subscription or other map visualization in Power BI, go for it!

For me, this version of the ArcGIS Map also reminded me that I needed to purchase a National Geographic Magazine subscription for my daughter :-)

Happy data adventures!
(2018-July-25) I like the ESRI company slogan, "The Science of Where". Hopefully, it will help someone not to get lost in the myriads of geo points on their maps. 

Working with ArcGIS map visualization in Power BI, you always hit that notification point that tells you about more features available in the non-free version of the ArcGIS map, so I wanted to explore this.

I pulled an opened dataset of TTC Routes and Schedules with the route definitions, stop patterns, stop locations, and schedules of the Toronto TTC ground buses and streetcars into my Power BI data model.

With the following table count statistics:
Stops - 10,614 records
Trips - 134,949 records
Routes - 200 records
Stop Times - 5,584,011 records

And this data model helped me to plot all (almost all) stops' geo locations in my ArcGIS map:

My next step was to sign up for the Plus subscription (Paid version of the ArgGIS map) in Power BI. The registration went smoothly, my credit card information has been collected, no warning signs. However, when I tried to sign in again and authenticate myself with my registration email address, I received an error message that something went wrong with ArcGIS Maps for Power BI with apologies for the inconvenience and request to refresh the page or check back later. I contacted the Canadian ESRI tech support but still haven't received any updates for this issue.

So, I was only left with the option to trust the ESRI documentation and check limitations of the free version of the ArcGIS Map in Power BI. If you read the documentation, basically you will get an idea of "more", there a couple of additional new features, but the Plus version of the map has all the standard features with more capacity:

Included with Power BIWith a Plus subscription
4 basic basemaps
4 basic basemaps and 8 others, including satellite imagery
1,500 features per map
5,000 features per map
100,000 features per month
1 million features per month
Reference layers
10 reference layers that contain U.S. demographics
Access to Esri Living Atlas maps and layers (feature services)
Publicly shared feature layers on ArcGIS
Publicly shared feature layers on ArcGIS
Curated gallery of U.S. demographics variables (7 categories)
Full access to the ArcGIS GeoEnrichment data browser, including U.S. and global demographics variables

More basic maps, more geo coordinates to map, more reference layers. The good thing about the reference layers is that you get exposed to the same set of publicly shared feature layers in both free and paid versions of the map, which is a cool thing! There is a search field in the Edit mode of the map that allows you to select and view one of those public layers within your map.

So now, knowing what my limits are, I switched my map theme to clustering and realized that it also had another limit that is not listed on the official ESRI site, the clustering feature in my case was limited to 10,000 geo points vs. the actual dataset of 10,614 stop locations. You can check those numbers if you like.

I still hope that ESRI tech support team will get back to me with my inquiry, otherwise, I will need to find a way to cancel my Plus subscription.

Happy data adventures!

(2018-July-29) Update: the ESRI support team finally reached out to me and helped to resolve the issue, please read my 2nd part of this story:
(2018-May-27) With recent Hawaiian volcano activity, it would be interesting to see how other similar events have shaped our Earth planet. The NOAA's National Centers for Environmental Information (NCEI) provides public access to the Volcano Location Database - And currently, it contains information about 1571 known volcanic eruptions.

I was able to extract this dataset into my Power BI data model and decided to use the new custom map visualization created by Mapbox that became available with this monthly Power BI update. 

The working experience with Mapbox is similar to other custom and native map visualization, however, it has one new additional feature that would require you to register and get your personal Mapbox access token.

As soon as you place this visualization control into your report canvas and then link it with your geo dataset, the following initial screen will show up:

Then after the initial steps of registration, you are able to get this access token to place into the Power BI Viz format panel:

Going back to the actual volcanic map, I found the proof, that the Ring of Fire does exist and it's around the Pacific Ocean.

And here is the actual Power BI report with the volcanic interactive map using Mapbox (which currently is only supported in Chrome and Firefox) :

Happy data adventures!
(2018-May-20) A childhood dream to travel around the world fueled by reading Gulliver's Travels stories and Robinson Crusoe attempts to survive on a deserted island. Those books were filled with geographical description and map locations. I think I always loved working with maps and I still do! There is something that could captivate your mind and propel your imagination for traveling when you immerse yourself in maps.

Power BI itself a great tool that could feed your appetite for an adventure using built-in and custom map visualizations.

The built-in (or native) map visualizations are available right away when you install the Power BI desktop for the first time and Microsoft has provided sufficient information for you to get started working with them:
- Map
- Filled Map
- ArgGIS Map
- Shape Map

And there is another set of custom map visualization that we can use in building data analytics in Power BI (not a complete list):
- Flow map by Weiwei Cui
- Heatmap by Weiwei Cui
- Globe Map by Microsoft 
- Icon Map by Altius
- Mapbox Visual by Mapbox

Since there is more information about the built-in map visualizations then I'd like to share my experience working with the custom ones.

But before I do this, let me spend a few minutes describing how the Shape Map (still in Preview at the moment) works.

Shape Map

Once you enable this Preview feature in your desktop version of the Power BI then can you use a predefined set of Shapes:

or you can free your mind and upload a custom shapefile with Canadian provinces and Northen Territories, which I did from this site

Flow map by Weiwei Cui

Flow map is a special type of network visualization that portrays movements of objects among geo-locations. 

I built a simple dataset with the list of Provinces and Territories capitals and linked them all to Ottawa (Canadian capital). The component itself has a very rich set of formatting settings which I liked the most; so the final visualization could look totally different by changing one of them.

Heatmap by Weiwei Cui

Heat maps are a type of visualization to show data density on a map. They are particularly helpful when you have a lot of (e.g., tens of thousands of) data points on the map and are mainly interested in their overall distribution. Technically, in a heatmap, data points are aggregated locally and mapped to colors (either gradient or quantile), so that we can make better sense of the density of the data from the colors while still being able to see and use the map. 

I already blogged about the use of the heatmaps in Power BI - and we can always use the built-in ArgGIS heat maps option as well to see alternative ways for your dataset density.

Globe Map by Microsoft

One the best features of the GlobeMap is that it allows you to rotate the Globe and see it from different angles, and this is right within your Power BI report.

Globe Map is a 3D Map that makes the map exploration experience more immersive and magical. It provides the sense of connection to the data with the physical world. This, combined with our spatial ability, brings a new perspective to the data when presented as 3D objects. Globe Map also supports heat map on the spatial map. You can use a second measure for heat intensity and draw immediate attention to the right areas.

I used the Berkley Earth datasets - for this 3D reporting which was already featured in my previous blog post - 

Icon Map by Altius

The Icon Map visual shows data-bound images & lines or circles on a map and allows you to render images and lines or circles on the map. 

Mapbox Visual by Mapbox

The Mapbox makes heatmaps, circles, and clusters using big location data.
Your location data is more than just points on a map. With the Mapbox Visual for Power BI, quickly create beautiful custom map visuals that answer your business question with fast performance.

And here is the actual Power BI report with some of the custom map visualizations:

Happy data adventures!