Merge pull request #4 from CompNet/dev

Version 1.0.2

Author: LucasPotin98

README.md

@@ -1,24 +1,24 @@
-FoppaInit v1.0.1
+FoppaInit v1.0.2
 -------------------------------------------------------------------------
 *Initialization of the FOPPA database*
 
-* Copyright 2021-2022 Lucas Potin & Vincent Labatut
+* Copyright 2021-2023 Lucas Potin & Vincent Labatut
 
 FoppaInit is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation. For source availability and license information see `licence.txt`
 
 * **Lab site:** http://lia.univ-avignon.fr
 * **GitHub repo:** https://github.com/CompNet/FoppaInit
-* **Data:** https://doi.org/10.5281/zenodo.7443842
+* **Data:** https://doi.org/10.5281/zenodo.7808664
 * **Contact:** Lucas Potin <lucas.potin@univ-avignon.fr>
 
 -------------------------------------------------------------------------
 
 # Description
-These scripts create the FOPPA database v.1.1.1 from raw TED files. This database relies mainly on the award notices of public contracts related to French clients and suppliers from 2010 to 2020 in the Tenders Electronic Daily. It also proposes an enrichment of these data, thanks to the siretization of agents (i.e. the retrieval of their unique IDs, which is missing for most of them) as well as the cleaning and extraction of award criteria, and other processing.
+These scripts create the FOPPA database v.1.1.2 from raw TED files. This database relies mainly on the award notices of public contracts related to French clients and suppliers from 2010 to 2020 in the Tenders Electronic Daily. It also proposes an enrichment of these data, thanks to the siretization of agents (i.e. the retrieval of their unique IDs, which is missing for most of them) as well as the cleaning and extraction of award criteria, and other processing.
 
-The process conducted to build the FOPPA is quite long, though (around 1 week, depeding on the hardware), so the produced database is alternatively directly available on [Zenodo](https://doi.org/10.5281/zenodo.7443842). The detail of this processing are described in a technical report [P'22].
+The process conducted to build the FOPPA is quite long, though (around 1 week, depeding on the hardware), so the produced database is alternatively directly available on [Zenodo](https://doi.org/10.5281/zenodo.7808664). The detail of this processing are described in an article [[P'23]](#references), and in further detail in a technical report [[P'22]](#references).
 
-This work was conducted in the framework of the [DeCoMaP](https://anr.fr/Projet-ANR-19-CE38-0004) ANR project (*Detection of corruption in public procurement markets* -- `ANR-19-CE38-0004`). If you use this source code or the produced database, please cite bibliographical reference [P'22].
+This work was conducted in the framework of the [DeCoMaP](https://anr.fr/Projet-ANR-19-CE38-0004) ANR project (*Detection of corruption in public procurement markets* -- `ANR-19-CE38-0004`). If you use this source code or the produced database, please cite bibliographical reference [[P'23]](#references) (the article, not the report).
 
 # Organization
 This repository is composed of the following elements:
@@ -51,7 +51,7 @@ In order to build the FOPPA database:
 
 The script is going to perform several tasks:
 1. Download all the necessary data (see Section *Organization*).
-2. Apply the processing described in [P'22].
+2. Apply the processing described in [[P'22]](#references).
 3. Export the resulting database under different forms (SQL dump, CSV sheets).
 
 # Dependencies
@@ -70,6 +70,5 @@ The produced database is directly available publicly online on [Zenodo](https://
 * CSV files (one by table).
 
 # References
-**[P'22]** L. Potin, V. Labatut, R. Figueiredo, C. Largeron & P. H. Morand. *FOPPA: A database of French Open Public Procurement Award notices*, Technical Report, Avignon Université, 2022. [⟨hal-03796734⟩](https://hal.archives-ouvertes.fr/hal-03796734)
-
-
+* **[P'23]** L. Potin, V. Labatut, P. H. Morand & C. Largeron. *FOPPA: An Open Database of French Public Procurement Award Notices From 2010–2020*, Scientific Data, 2023, 10:303. DOI: [10.1038/s41597-023-02213-z](https://dx.doi.org/10.1038/s41597-023-02213-z) [⟨hal-04101350⟩](https://hal.archives-ouvertes.fr/hal-04101350)
+* **[P'22]** L. Potin, V. Labatut, R. Figueiredo, C. Largeron & P. H. Morand. *FOPPA: A database of French Open Public Procurement Award notices*, Technical Report, Avignon Université, 2022. [⟨hal-03796734⟩](https://hal.archives-ouvertes.fr/hal-03796734)

foppaInit.py

@@ -237,13 +237,13 @@ def databaseCreation(nameDatabase):
     sql = cursor.execute(request)
     request = "CREATE TABLE CriteriaTemp (lotId INTEGER,CRIT_PRICE_WEIGHT TEXT,CRIT_WEIGHTS TEXT, CRIT_CRITERIA TEXT)"
     sql = cursor.execute(request)
-    request = "DROP TABLE IF EXISTS LotClients"
+    request = "DROP TABLE IF EXISTS LotBuyers"
     sql = cursor.execute(request)
-    request = "CREATE TABLE LotClients(lotId INTEGER,agentId INTEGER,FOREIGN KEY(agentId) REFERENCES Agents(agentId) ON UPDATE CASCADE,FOREIGN KEY(lotId) REFERENCES Lots(lotId) ON UPDATE CASCADE)"
+    request = "CREATE TABLE LotBuyers(lotId INTEGER,agentId INTEGER,FOREIGN KEY(agentId) REFERENCES Agents(agentId) ON UPDATE CASCADE,FOREIGN KEY(lotId) REFERENCES Lots(lotId) ON UPDATE CASCADE)"
     sql = cursor.execute(request)
-    request = "DROP TABLE IF EXISTS LotSuppliers"
+    request = "DROP TABLE IF EXISTS LotWinners"
     sql = cursor.execute(request)
-    request = "CREATE TABLE LotSuppliers(lotId INTEGER,agentId INTEGER,FOREIGN KEY(agentId) REFERENCES Agents(agentId) ON UPDATE CASCADE,FOREIGN KEY(lotId) REFERENCES Lots(lotId) ON UPDATE CASCADE)"
+    request = "CREATE TABLE LotWinners(lotId INTEGER,agentId INTEGER,FOREIGN KEY(agentId) REFERENCES Agents(agentId) ON UPDATE CASCADE,FOREIGN KEY(lotId) REFERENCES Lots(lotId) ON UPDATE CASCADE)"
     sql = cursor.execute(request)
     request = "DROP TABLE IF EXISTS Names"
     sql = cursor.execute(request)
@@ -384,7 +384,7 @@ def firstCleaning(datas,database):
                 VALUES (?,?,?,?,?,?,?,?,?)'''
             val = (compteurAgent,tempName,tempSiret,tempAddress,tempTown,tempPC,tempCountry,date,"CAE")
             cur.execute(sql,val)
-            sql = ''' INSERT INTO LotClients(lotId,agentId)
+            sql = ''' INSERT INTO LotBuyers(lotId,agentId)
                 VALUES (?,?)'''
             val = (i,compteurAgent)
             cur.execute(sql,val)
@@ -443,7 +443,7 @@ def firstCleaning(datas,database):
                     VALUES (?,?,?,?,?,?,?,?,?)'''
             val = (compteurAgent,tempName,tempSiret,tempAddress,tempTown,tempPC,tempCountry,date,"WIN")
             cur.execute(sql,val)
-            sql = ''' INSERT INTO LotSuppliers(lotId,agentId)
+            sql = ''' INSERT INTO LotWinners(lotId,agentId)
                     VALUES (?,?)'''
             val = (i,compteurAgent)
             cur.execute(sql,val)
@@ -1258,22 +1258,22 @@ def dedupeAgent(datas,database):
 
 def finalTableAgent(database):
     cursor = database.cursor()
-    clients = pd.read_sql_query("SELECT * FROM LotClients", database,dtype=str)
-    suppliers = pd.read_sql_query("SELECT * FROM LotSuppliers", database,dtype=str)
+    clients = pd.read_sql_query("SELECT * FROM LotBuyers", database,dtype=str)
+    suppliers = pd.read_sql_query("SELECT * FROM LotWinners", database,dtype=str)
     names = pd.read_sql_query("SELECT * FROM Names", database,dtype=str)
     datas = pd.read_csv("ResDedupe.csv",dtype=str,sep=",")
     lenCluster = len(datas.groupby("Cluster ID").count())
     request = "DROP TABLE IF EXISTS Agents"
     sql = cursor.execute(request)
     request = "CREATE TABLE Agents(agentId INTEGER,name TEXT,siret TEXT,address TEXT,city TEXT,zipcode	TEXT,country TEXT, department TEXT,longitude TEXT, latitude TEXT,PRIMARY KEY(agentId))"
     sql = cursor.execute(request)
-    request = "DROP TABLE IF EXISTS LotClients"
+    request = "DROP TABLE IF EXISTS LotBuyers"
     sql = cursor.execute(request)
-    request = "CREATE TABLE LotClients(lotId INTEGER,agentId INTEGER)"
+    request = "CREATE TABLE LotBuyers(lotId INTEGER,agentId INTEGER)"
     sql = cursor.execute(request)
-    request = "DROP TABLE IF EXISTS LotSuppliers"
+    request = "DROP TABLE IF EXISTS LotWinners"
     sql = cursor.execute(request)
-    request = "CREATE TABLE LotSuppliers(lotId INTEGER,agentId INTEGER)"
+    request = "CREATE TABLE LotWinners(lotId INTEGER,agentId INTEGER)"
     sql = cursor.execute(request)
     request = "DROP TABLE IF EXISTS Names"
     sql = cursor.execute(request)
@@ -1297,14 +1297,14 @@ def finalTableAgent(database):
 
     for i in range(len(clientsLot)):
         if (int(clientsAgent[i]) in dico):
-            sql = ''' INSERT OR IGNORE INTO LotClients(lotId,agentId)
+            sql = ''' INSERT OR IGNORE INTO LotBuyers(lotId,agentId)
                         VALUES (?,?)'''
             val = (int(clientsLot[i]),int(dico[int(clientsAgent[i])]))
             cursor.execute(sql,val)
 
     for i in range(len(suppliersLot)):
         if (int(suppliersAgent[i]) in dico):
-            sql = ''' INSERT OR IGNORE INTO LotSuppliers(lotId,agentId)
+            sql = ''' INSERT OR IGNORE INTO LotWinners(lotId,agentId)
                         VALUES (?,?)'''
             val = (int(suppliersLot[i]),int(dico[int(suppliersAgent[i])]))
             cursor.execute(sql,val)
@@ -1522,10 +1522,10 @@ def exportDatabase(database):
     agents.to_csv("FOPPA/csv/Agents.csv",index=False)
     criteria = pd.read_sql_query("SELECT * FROM Criteria", database)
     criteria.to_csv("FOPPA/csv/Criteria.csv",index=False)
-    lotsClients = pd.read_sql_query("SELECT * FROM LotClients", database)
-    lotsClients.to_csv("FOPPA/csv/LotClients.csv",index=False)
-    lotsSuppliers = pd.read_sql_query("SELECT * FROM LotSuppliers", database)
-    lotsSuppliers.to_csv("FOPPA/csv/LotSuppliers.csv",index=False)
+    lotsClients = pd.read_sql_query("SELECT * FROM LotBuyers", database)
+    lotsClients.to_csv("FOPPA/csv/LotBuyers.csv",index=False)
+    lotsSuppliers = pd.read_sql_query("SELECT * FROM LotWinners", database)
+    lotsSuppliers.to_csv("FOPPA/csv/LotWinners.csv",index=False)
     names = pd.read_sql_query("SELECT * FROM Names", database)
     names.to_csv("FOPPA/csv/Names.csv",index=False)
 
@@ -1599,4 +1599,4 @@ def exportDatabase(database):
     db.close()
     #os.remove("Foppa.db")
     del db
-    
+