how to call fabric rest apis from fabric data pipelines.
introduction.
When writing the article on how to call power bi rest apis from fabric data pipelines, Fabric itself was still in preview. In fact, it became GA only one day later, with which the Fabric specific APIs were also released (well in fact, they were pushed to public preview). Since Fabric REST APIs need a slightly different setup than Power BI REST APIs, I thought I’d write a separate article on how to get started. After all, there seems to be some overlap, i.e. the get capacities call delivers rather similar though not exactly the same information. I reckon Fabric REST APIs will rapidly expand in the future: Many new calls will be added to go in-step with Fabric as a whole, surpassing the capabilities of Power BIs REST APIs. Though I am not sure, whether they will add all of Power BI’s APIs to the API suite of Fabric or whether they will keep them separate. This topic is probably debated on many other levels, too. Time will show :)
prerequisites.
1. A Fabric capacity and workspace
2. A Data pipeline
1. What’s the goal?
The goal is to be able to call Fabric REST APIs from a Fabric Data Pipeline. As seen in the example below, we called the Get Workspace API returning a workspace.
2. Set up a new connection
At first, we need to set up a new Web V2 connection that will be used to call the API. For this, click on the settings icon in the top right and then Manage connections and gateways.
Next, click on + New
Now, select the Cloud option, provide a name and fill in the other fields according to the picture. When you are done, press the Create button.
Connection type: Web V2
Base Url:https://api.fabric.microsoft.com/v1
Token Audience Url: https://api.fabric.microsoft.com
Note, the Token Audience Url is something I have not yet found in the documentation. I just tried a few things until I got one that worked. Once again, the API is still in preview and this could be subject for change. There might even be another Audience Url recommended by Microsoft.
For the Authentication method, I selected OAuth 2.0. You need to provide your credentials by clicking on Edit credentials in order to be able to create the connection. Here you log in as a user, and then the data pipeline authenticates to Fabric on your behalf.
In a real life example, I highly recommend to use a service principal instead of the OAuth 2.0 method. In the appendix, I listed the steps necessary for using a service principal for authentication.
A couple of considerations on this matter: I tried a few different APIs and some of them did not work with a service principal but only with the OAuth2 method (i.e. the Lakehouse ones did not work with a service principal). In that case, I got the error message ““PrincipalTypeNotSupported. The operation is not supported for the principal type“.
I am not sure if they disabled those APIs for service principals or whether they just have not gotten there yet to enable them. Neither do I know whether Microsoft will activate those at some point or whether this is by design. Anyway, Microsoft does claim that service principal do not work with Fabric APIs as a (current?) limitation. Interesting enough, Microsoft explicitly states that the Power BI REST APIs can be used with service principals. My guess is they will come. Some of them already work with service principals after all.
3. Add Web Activity to your Fabric Data pipeline
Now, add a web activity, pick the recently created connection, fill in “/workspaces” for the Relative URL and you are good to go. Of course, for other APIs you need to align the Relative URL and for more advanced APIs like POST or PUT, you also need to fill the body. By the way, for the error “The request entity’s media type ‘text/plain’ is not supported for this resource”, try adding a header with the name content-type and the Value application/json.
4. Showtime
When running the pipeline with the above settings, the workspace is returned in the output. This call worked for both OAuth2.0 and service principal as authentication methods.
As mentioned before, when using a service principal calling the List Table API for the lakehouse, it threw an error:
With the OAuth2.0 authentication, it did work:
end.
I am really looking forward to see what will be added to the API repository. Looking at the list of scopes already laying in the API Permissions, it seems there is loads in the pipe. Of course, not all of them will be appropriate to call from a data pipeline but certainly some will. Excited to see what people come up with to use them in data pipelines. Finally, you could also fire those APIs from Python notebooks and, in turn, call those notebooks from your Fabric Data Pipeline.
appendix.
If you wanna use a service principal for authenticating to Fabric/Power BI, you first need to create a service principal in Azure. For this, go into the Azure Portal and create a new app registration. Note, there is no need to setup any scopes/API permissions.
Now, paste the Tenant ID, the Service principal ID and the Service principal key into the boxes:
The Tenant ID (or Directory ID) can be found in the Overview tab of your service principal. The Service principal ID equals the Application or Client ID:
The Service principal key is a secret you need to create under Certificates & secrets:
Also, you have to allow service principals to use REST API calls. You can find and enable this setting in the Admin portal in Fabric. Note, it is probably a good idea to not apply this configuration to the entire organization. It is better to use dedicated security groups for such matters.
And lastly, we need to add the service principal to our fabric workspace at least as a contributor:
