How to Export Commit and Pull Request Data from Bitbucket to CSV

November 26, 2020
#Bitbucket#Reporting#How To
11 min

Being a universal file type, CSV serves as a go-to format for integrations between the applications. It allows for transferring a large amount of data across the systems, even if the integration is not supported natively.  However, you can’t export commit and pull request data from Bitbucket out of the box. The good news is that Awesome Graphs for Bitbucket gives you the capability to export to CSV in different ways.

In this article, we’ll show you how you can use the app to export engineering data to CSV for further integration, organization, and processing in analytics tools and custom solutions.

What you will get

The described ways of exporting will give you two kinds of generated CSV files, depending on the type of data exported. 

In the case of commit data, you’ll get a list of commits with their details:

list of commits with details

And the resulting CSV with a list of pull requests will look like this:

export commit and pull request data to csv

Exporting from the People page

You can export raw commit and pull request data to CSV directly from Bitbucket. When you click All users in the People dropdown menu at the header, you’ll get to the People page with a global overview of developers’ activity in terms of commits or pull requests.

At the top-right corner, you’ll notice the Export menu, where you can choose CSV.

export pull request data from bitbucket

By default, the page shows contributions made within a month, but you can choose a longer period up to a quarter. The filtering applies not only to the GUI but also to the data exported, so if you don’t change the timespan, you’ll get a list of commits or pull requests for the last 30 days.

Exporting via the REST API resources

Beginning with version 5.5.0, Awesome Graphs REST API allows you to retrieve and export commit and pull request data to CSV on global, project, repository, and user levels, using the dedicated resources. This functionality is aimed to automate the processes you used to handle manually and streamline the existing workflows.

You can access the in-app documentation (accessible to Awesome Graphs’ users) by choosing Export → REST API on the People page or go to our documentation website.

We’ll show you two examples of the resources and how they work: one for exporting commits and another for pull requests. You’ll be able to use the rest of the resources as they follow the model.

Export commits to CSV

This resource exports a list of commits with their details from all Bitbucket projects and repositories to a CSV file.

Here is the curl request example:

curl -X GET -u username:password "https://bitbucket.your-company-name.com/rest/awesome-graphs-api/latest/commits/export/csv" --output commits.csv

Alternatively, you can use any REST API client like Postman or put the URL directly into your browser’s address bar (you need to be authenticated in Bitbucket in this browser), and you’ll get a generated CSV file.

By default, it exports the data for the last 30 days. You can set a timeframe for exported data up to one year (366 days) with sinceDate / untilDate parameters:

curl -X GET -u username:password "https://bitbucket.your-company-name.com/rest/awesome-graphs-api/latest/commits/export/csv?sinceDate=2020-10-01&untilDate=2020-10-13" --output commits.csv

For commit resources, you can also use the query parameters such as merges to filter merge/non-merge commits or order to specify the order to return commits in.

Read more about the resource and its parameters.

Export pull requests to CSV

The pull request resources work similarly, so to export a list of pull requests with their details from all Bitbucket projects and repositories to a CSV file, make the following curl request:

curl -X GET -u username:password "https://bitbucket.your-company-name.com/rest/awesome-graphs-api/latest/pull-requests/export/csv" --output pullrequests.csv

The sinceDate / untilDate parameters can also be applied to state the timespan up to a year, but here you have an additional parameter dateType, allowing you to choose either the creation date or the date of the last update as a filtering criterion. So, if you set dateType to created, only the pull requests created during the stated period will be returned, while dateType set to updated will include the pull requests that were updated within the time frame.

Another pull request specific parameter is state, which allows you to filter the response to only include openmerged, or declined pull requests.

For example, the following request will return a list of open pull requests, which were updated between October 1st and October 13th:

curl -X GET -u username:password "https://bitbucket.your-company-name.com/rest/awesome-graphs-api/latest/commits/export/csv?dateType=updated&state=open&sinceDate=2020-10-01&untilDate=2020-10-13" --output pullrequests.csv

Learn more about this resource.

Integrate intelligently

While CSV is supported by many systems and is quite comfortable to manage, it is not the only way for software integrations the Awesome Graphs for Bitbucket app offers. Using the REST API, you can make the data flow between the applications and automate the workflow, eliminating manual work. And we want to make it easier for you and save your time.
Let us know what integrations you are interested in, and we’ll try to bring them to you, so you don’t have to spend time and energy creating workarounds.

Integration of Yellowfin and Confluence

April 21, 2015
#How To#Confluence
13 min

The modern world abounds with enormous amounts of data that remain just characters and digits without a tool that can aggregate, process and visualize this data for better perception of information. Yellowfin is an analytic software solution that allows you to maximize efficiency of your business intelligence with numerous reports and dashboards. It supports multiple data sources, such as relational databases, multi-dimensional cubes or in-memory analytical databases.

In this blog post we will review integration between Yellowfin reporting platform and Atlassian Confluence for quicker team collaboration.

Standard Integration Capabilities

Yellowfin is equipped with JavaScript API, which allows you to embed JavaScript code snippets onto your web pages. You can embed either reports or dashboards onto your pages with the JS code snippets automatically generated in Yellowfin.

Yellowfin and Confluence

A generic code snippet will look like the following one:

//JS code to embed the report
<script type="text/javascript" src="http://yellowfin_server/JsAPI?reportUUID=370cf271-0b78-4e55-ba81-cf4a71e29acf&width=829&height=295"></script>

//JS code to embed the dashboard
<script type="text/javascript" src="http://yellowfin_server/JsAPI?dashUUID=9a31442d-d8a6-4aad-ad11-d342db606b64&width=800&height=400"></script>

The basic script includes address of the Yellowfin server, path to JavaScript API and the size of the reporting area in pixels. As usual, you will have to include the login and password in the JS code if you want to embed it into some external resource.

//JS code to embed the report
<script type="text/javascript" src="http://yellowfin_server/JsAPI?reportUUID=370cf271-0b78-4e55-ba81-cf4a71e29acf&width=829&height=295&username=admin@yellowfin.com.au&password=test"></script>

So what can you do with this JavaScript code in Confluence? You can create a new page and add HTML macro, which you can further paste your report or dashboard code into.

Yellowfin and Confluence

Once you save the page, you will be required to enter the credentials once again and then the report will be generated.

Yellowfin and Confluence

You can embed multiple reports per Confluence page and get data visualized in one place.

This way of embedding Yellowfin data is not very convenient as it exposes your credentials and all the time you need to verify JavaScript code as you can make any mistake in it.

Inserting Yellowfin Reports via User Macro

A better way to embed Yellowfin reports and dashboards into your Confluence pages is to write a user macro with the interface for entering parameters.

Navigate to Administration console > Configuration and select User macros. Once the page opens, click Create a user macro.

On the opened page, specify the basic information about the user macro. You need to enter the user macro name which will be used for inserting the macro on the page, define its visibility and select the category, which the your user macro will be associated with.

Yellowfin and Confluence

For the Macro Body Processing, select ‘Rendered’ so Confluence will render the content received from Yellowfin API.

Yellowfin and Confluence

As the macro template enter the following code:

## @param elementId:title=Element ID|type=string|required=true|desc=Enter the Element iD
## @param reportUUID:title=UUID|type=string|required=false|desc=Enter the report UUID
## @param showFilters:title=Show Filters|type=boolean|required=false|desc=Enable the information showing
## @param showExport:title=Show Export|type=boolean|required=false|desc=Enable display of page links
## @param height:title=Height|type=int|required=false|desc=Enter height of the report
## @param width:title=Width|type=int|required=false|desc=Enter height of the report
 
<ac:structured-macro ac:name="html">
    <ac:parameter ac:name="atlassian-macro-output-type">INLINE</ac:parameter>
    <ac:plain-text-body><![CDATA[<div id="$paramelementId">

//Both Confluence and Yellowfin are installed on the local computer. If instances are installed on different servers enter the appropriate address of Yellowfin server.
<script src="http://127.0.0.1:8087/JsAPI" type="text/javascript"></script>
<script type="text/javascript">
	var options = {};
		options.elementId = '$paramelementId';
		options.reportUUID = '$paramreportUUID';
		options.username = 'youremail@example.com';
		options.password = 'yourpassword';
		options.showFilters = '$paramshowFilters';
		options.showExport = '$paramshowExport';
		options.width = '$paramwidth';
		optionsheight = '$paramheight';
	yellowfin.loadReport(options);
</script>

</div> </ac:plain-text-body>

So let’s divide this code into snippets and see what each piece of it does.

This code snippet defines the set of input elements available in the macro browser. For each input element you need to define its name, title in the macro browser, type, required or not, and text description of the input element. You can find detailed information about parameters in user macros in Confluence documentation.

## @param elementId:title=Element ID|type=string|required=true|desc=Enter the Element iD
## @param reportUUID:title=UUID|type=string|required=false|desc=Enter the report UUID
## @param showFilters:title=Show Filters|type=boolean|required=false|desc=Enable the information showing
## @param showExport:title=Show Export|type=boolean|required=false|desc=Enable display of page links
## @param height:title=Height|type=int|required=false|desc=Enter height of the report
## @param width:title=Width|type=int|required=false|desc=Enter height of the report

The screenshot below illustrates these input elements in the macro browser.

Yellowfin and Confluence
The next code snippets is our macro body that inserts the HTML macro that will be used for output of reports and dashboards from Yellowfin server.

//Inserts the HTML macro that will be used to wrap our Yellowfin script.
<ac:structured-macro ac:name="html">
    <ac:parameter ac:name="atlassian-macro-output-type">INLINE</ac:parameter>
    <ac:plain-text-body><![CDATA[<div id="$paramelementId">

//Calls Yellowfin script that will fetch the report or dashboard from Yellowfin server.
<script type="text/javascript">
	YELLOWFIN SCRIPT GOES HERE
</script>

</div> </ac:plain-text-body>

We want to point your attention to the specific HTML element that will be used for output of Yellowfin report/dashboard stored in the tag shown below. Our user macro will allow you to enter its ID right in the macro browser, so you will be able to insert multiple reports and dashboards on the same page without any issues. The $paramelementId parameter will take the value entered as Element ID and paste it into our user macro.

<ac:plain-text-body><![CDATA[<div id="$paramelementId">
</div> </ac:plain-text-body>

And the last part of out user macro is the call of Yellowfin script that will make a request to its JavaScript API on the server and fetch the required report with the defined parameters. As you can see $param+parameterName is used to fetch the value defined in the macro browser and insert it into the script. In Yellowfin documentation you can find the fullt list of parameters for reports and dashboards. The example below provides a range of basic parameters required for output of the report, the similar code can be used for dashboards. You can alternate the parameters or remove the unnecessary ones if needed. Some parameters are mutually exclusive so please consult Yellowfin API documentation before you start.

<script src="http://127.0.0.1:8087/JsAPI" type="text/javascript"></script>
<script type="text/javascript">
	var options = {};
		options.elementId = '$paramelementId';
		options.reportUUID = '$paramreportUUID';
		options.username = 'youremail@example.com';
		options.password = 'yourpassword';
		options.showFilters = '$paramshowFilters';
		options.showExport = '$paramshowExport';
		options.width = '$paramwidth';
		optionsheight = '$paramheight';
	yellowfin.loadReport(options);
</script>

Once you complete save the user macro.

Inserting User Macro

Open the necessary Confluence page and switch it to the edit mode.

Start entering {Ylbrep} and select your user macro. Name of your user macro may vary if you set a different one.

Yellowfin and Confluence

Then just enter the required Element ID and insert report UUID from Yellowfin server and that’s it. If needed, enable display of filters and report export, and specify the required width and height of the report.

Once you save the macro, its parameters will be saved in its body.

When you save the page the report will appear on it and you can work with it as in Yellowfin itself. The report will be updated upon each reload of Confluence page, so if new data is continuously supplied to Yellowfin you will get all the time the updated data.

Yellowfin and Confluence

As you can see it is not a problem to quickly and easily integrate Yellowfin and Confluence. Yellowfin Javascript API is easy to understand, use and customize. Besides the generic embedding of the report or dashboard, you can insert them with the already filtered results and save a bit of your time.