Ebay to Shopify Data Extractor / Scraper

eBay Extractor / Scraper
Java desktop application using Google Chrome to extract eBay item data.

Note: this post was previewed and approved by the customer.

Completion date: February 2015.

In this project, we wrote a Java desktop application to control Google Chrome and extract item data from eBay. Through this method we achieved 100% accuracy.

The “Search URL” input will take a list of eBay search URLs. Clicking on the “Get Item Numbers” button will scrape each search URL and its next results pages, then add the item numbers found to the next input box.

You can add, edit or remove any item numbers in this box. You can even use it without scraping a search page first.

Finally, selecting an output file and clicking on the “Extract” button will extract data from each item page and populate a file in CSV format suitable for Shopify import.

If you need something similar, feel free to contact us.


Agile management project: Python / wxPython based realistic test data generator for databases (III): selecting developers

Note: this post was previewed and approved by the customer.

In order to post a bid on behalf of the employer in vWorker (adquired by Freelancer on November 2012), he must give me subaccount permissions in his control panel. This is pretty straightforward, and the employer has control over the permissions I will have in his account. To do this, the employer needs to log in to vWorker, go to My account → My subaccounts, click on Add new sub-account and then follow the instructions there.

Once this is done, I can log in to the employer’s subaccount. I use several browsers, one with my main account and others with different subaccounts, so I don’t need to switch the account constantly.

Now it’s time to post a new project by clicking on the Post a new project link at the top left of the page. It’s important to be clear and concise about what you need. On freelance sites like vWorker, a developer needs to place a lot of bids to win a single project, so it’s better to write a small specification on the bid page, and put a full specification in an attachment. Here is a screenshot of the new project page:

vWorker New Project Page
Full page screenshot of the vWorker new project page. Click to enlarge.

From four interested candidates, we selected two developers who were in a nearby timezone, communicated well, offered us a reasonable price, and had demonstrable experience with wxPython development.

We started with a small test (fixing a minor bug) using pay for deliverables. In this way the developer has a good idea of what the work will involve and we can know more about his work before starting the bulk of the project. These tests were successful, so we switched to pay for time, as we consider it better suited to agile development than pay for deliverables.

Agile management project: Python / wxPython based realistic test data generator for databases (II): defining an infrastructure

Note: this post was previewed and approved by the customer.

an infrastruture
An infrastructure (Soo Locks, source: http://en.wikipedia.org/wiki/Soo_Locks)

The customer already provided a complete high level project specification, so I will begin defining lower level requirements. First I will define an infrastructure for the project. This is a desktop application, which means defining infrastructure for it is usually simpler than it would be for a server application. We will use:

We divided this project into several iterations. In the first one we will use just two developers, so I will use just a few agile techniques (no product / sprint backlog and agile metrics for now).

I have found that many developers dislike the idea of using a heavyweight integrated development environment and prefer lightweight editors like notepad++, vi or emacs. But after many years as developer, I have found that a fully featured IDE saves a lot of time with features like code autocompletion, refactoring or the ability to show errors without having to compile the application.

Coding conventions are used to standardize the code structure and for improving code quality. Many developers prefer to do the coding “their own way”, but sometimes that is not an option when working in a team.

Basecamp will be useful for discussing high level features and milestones, as it provides a lot of collaborative tools customized to distributed software development: white boards, forums, project schedulers and trackers…

Low level development tasks will be managed via Trac, as it integrates pretty well a version control software (which allows to share the source code and see what each developer changed), a bug tracking system (where the issues and low level task are managed) and a wiki (what is useful for writing both internal and external documentation collaboratively). In the next articles I will explain more in detail how these work, and how I will use them in this project.

We will also use vWorker for selecting developers, and we will initially pay for deliverables, so we have a fixed cost. If the developers does a good work in the initial part we will consider using pay for time, as it give us more flexibility and it’s better suited for agile development.

Agile management project: Python / wxPython based realistic test data generator for databases (I)

Note: this post was previewed and approved by the customer.

I was recently hired as a manager via vWorker (adquired by Freelancer on November 2012) for improving a realistic test data generator for databases written in Python / wxPython. We already have a working prototype, and we are going to convert it into a commercial application.

A partial screenshot of the initial prototype
A partial screenshot of the initial prototype

After asking permission from the customer, I will describe here the steps and techniques that I will use to manage this project. I’m new to Python and wxPython, but the existing code is clear, so I don’t have any substantial trouble following it.

So, if you are curious about how a project is managed in vWorker using agile techniques, and you want to follow a real case, don’t forget to use one of the subscribe links in the right menu. Feel free to ask any questions or post a comment.

vWorker invites me to the new Tech Sherpa program

The vWorker.com site logo
vWorker logo

…and this is really great as they just invited 4 people for this new feature. So, what is a Tech Sherpa? From the vWorker latest news blog:

A Tech Sherpa™ is a cross-discipline expert with years of success both “climbing” (doing technical work) as well as “guiding” (managing others who do that work). A Tech Sherpa™ can take over some or all aspects of managing your project, and free you to focus on more productive things. Prices range from $25-$95/hour, depending on their skills and location in the world.

So if you want somebody to manage your project, including writing specs, hiring people, managing them, and testing the work, now vWorker offers you everything of these plus a employer triple-point money-back guarantee and a workforce of more than 300.000 skilled people. You can create an account (completely free) here. And if you want to create a project just for me, just click here.

You can learn more about what a Tech Sherpa is and the new project management feature in the vWorker site.

Update December 1, 2012. Freelancer.com acquired vWorker on October 19, 2012. So if you want to contact me, click on this link.