Frontpage

Search result

Showing results 1171 - 1180 of 1751

High performance computing

If you need high computing power for data analysis, modelling or machine learning, High Performance Computing (HPC) will give you access to an analysis platform that can handle complex calculations quickly and efficiently., HPC is a solution where your project is executed on a super computer in an HPC centre. Unlike traditional servers, HCP allows you to scale the computing power up or down according to your needs., Advantages of HCP:, High-performance:, Calculate and analyse large data volumes much faster than on regular servers., Adaptability:, Scale your capacity according to the project requirements., Storage:, HPC can store large volumes of data and quickly retrieve these from the memory., Not all projects require HCP. If you are working with small data volumes, which can be processed efficiently on a traditional server, a hosted server may be a more cost-efficient solution. If your analyses require extensive data processing, complex simulations or machine learning models, HPC will serve you well. To be able to use HPC for projects with data from Statistics Denmark, the HPC center in question must have entered an agreement with Statistics Denmark., Multiple HPC solutions – which one suits your needs?, Statistics Denmark offers three different solutions, which are outlined below. Read more, so you can choose the best solution., The API-solution, This solution is available to anyone who has an approved project in Denmark’s Data Portal (DDP) and an agreement with one of the HPC centers under the solution. You can find an updated list of the HPC centers included further down this page., With this solution, the user can decide whether data should be stored , both, on Research Services servers or a hosted server , and, the HPC center, or , only, at the HPC center. It is also possible to have some users associated with the project , only, in DDP, while others are linked to the project both in DDP , and, at the HPC center. This solution also allows you to connect HPC resources throughout the entire or parts of the project’s lifetime., NGC Solution, This solution is available for health related projects, as this is a requirement from NGC. The project must also be located on NGC for the entire duration of the project. An advantage with this solution is that you make use of SSPE (see below)., Shared Secure Processing Environment (SSPE) Solution, This solution is developed for projects that have their own additional data that needs to be linked to the project and where the data either:, is very large and therefore difficult and costly to send to Research Services for de-identification, or, for legal reasons, cannot be transferred to Research services servers., This solution means that Research Services can process the data directly on NGC’s servers. Read more about this further down the page., Link HPC solutions to your project via the API solution, It is now possible to link a project in the Denmark’s Data Portal to an HPC center for a short or long period. Below, you can find answers to questions regarding the solution., Which HPC centers are included?, Statistics Denmark has entered into agreements with the following HPC centers:, Computerome (DTU), Read more at Computerome, GenomeDK (AU), Read more at GenomeDK, We expect more centers will be added to the solution, so the list may change over time., What does it cost to set up?, For the setup, Research Services invoices five hours according to the current hourly rate. In addition, billing for data extraction, consultation, disk usage, processing of additional data etc. are charged according to our normal rates. , You can see our current hourly rates here, The five hours do not cover the HPC center's costs for setting up the project in their environment or their operating costs. If you have questions about the prices of HPC centers, please contact them directly., How does the API solution work with external data?, External data that needs to be merged with Statistics Denmark’s bank of basic data must first be sent to the Research Service for de-identification. The external data will then be transferred to the project environment at the HPC center., Which projects can use the program?, All projects that are either on Research Services servers or a hosted server can purchase this solution, provided they have entered into an agreement with the relevant HPC center., The project must have an approved project proposal before Research Services transfers data to the HPC area. Since this is an add-on service, you can prepare and submit your project proposal to Research Services before the agreement with the HPC center is finalized. The data you are provided with will only be extracted and transferred to the HPC center once the project proposal is approved and Research Services has received a copy of the written agreement between the institution and the HPC center. Contact the HPC center directly if you are interested in this type of agreement., Note, : Projects located at the National Genome Center (NGC) cannot use this solution., How to get started?, First, you must have an agreement with one or more HPC centers. This agreement is valid for the entire institution. Once the agreement with the HPC center is in place, you can prepare a list of which users on which projects should have access to which HPC center., The list should be sent to your contact person at Research Services, along with a copy of the agreement with the HPC center, who will ensure that access is granted to individual users under the relevant projects., Do I have access to Statistics Denmark’s bank of basic data on both the research machine and in the HPC environment?, It is up to the individual project whether Statistics Denmark’s bank of basic data should be stored both on the research machine (or a hosted server) and at the HPC center, or only at the HPC center., This means that some users on a given project can have access to both the research machine and HPC resources, while others only have access to the research machine. Please note that if you choose to have data in both environments, you will still need to pay the normal rate for disk usage on the research machine/operating costs on your hosted server., If the project should only process data in the HPC environment, it is not necessary to store data on the research machine. In this case, you can inform your contact person at Research Services that the copy of the data on the research machine can be deleted after a copy is transferred to the HPC area., The programs (scripts) used to extract data will be saved so they can be re-executed later if needed. In this case, a framework agreement is created to cover the hours Research Services use to re-deliver the data., Note, : It is a requirement that the project has an active server space either on DST’s own servers or a hosted server, as this is used to retrieve files. Additionally, login to the HPC environment must be done via the Denmark’s Data Portal. , HPC for health projects via the Danish National Genome Center (NGC), If you are working with data from Statistics Denmark and you have a project with a health-related purpose, we offer an HPC solution via the Danish National Genome Center (NGC). The HPC solution uses a , One-Node Architecture, , where calculations are carried out on one server at a time. The HPC center is located outside Statistics Denmark, but the project will be created and controlled by Statistics Denmark in the same way as other projects. You will still manage the project via Denmark’s Data Portal., To attach a new or existing project to NGC, you must:, Have created a supplementary agreement to your data processing agreement., (Re-)propose you project for approval with Research Services., Make an agreement with NGC., Be able to engage in dialogue with technical staff about the set-up of server access to NGC., Payment for use of the HPC center is settled directly with the center. For the use of NGC’s HPC infrastructure, you pay for installation, renting of hardware, operation and support., Further information about conditions for login with NGC and prices for installation, renting of hardware, support and operation, (pdf, in Danish), SSPE - New opportunity for Data Sharing through the HPC Solution for Health Projects, If you wish to work with very large volumes of data, on a project with a health-related purpose, Statistics Denmark now offers a new project setup through our High Performance Computing (HPC) solution at the Danish National Genome Centre (NGC)., The Shared Secure Processing Environment—known as the SSPE solution—allows third party data providers to make their data available within a closed environment which, in turn, enables users to analyze these data alongside data from Statistics Denmark. This makes it possible to support projects where the volume of the external data exceeds the storage and analyzing capacity of FSE's servers. Furthermore, it ensures that researchers bound by legal agreements not to transfer external data to third parties—including Statistics Denmark—can comply with those obligations., The SSPE solution is an advancement of the existing NGC setup. It is not meant as a replacement for the current setup, but as an additional offer to projects requiring greater analytical capacity. The same terms and conditions apply to projects wishing to use the SSPE solution as to those using the NGC solution., Contact and questions, Questions regarding the HPC center, , software in the HPC environment, prices for HPC resources, or setup/maintenance of the HPC environment: Contact the relevant HPC center directly., Questions regarding API and SSPE, Statistics Denmark, : Contact your contact person at Research Services or write to , forskningsservice@dst.dk, . ,

https://www.dst.dk/en/TilSalg/data-til-forskning/analyseplatform/high-performance-computing

Linking of additional data

Statistics Denmark’s register data can be linked with other data materials, here called additional data. This can be, for example, extractions from registers outside Statistics Denmark, your own data – for example survey data – or data from other data providers., Additional data must be documented and comply with the same requirements to data minimisation and statistical disclosure control that Statistics Denmark applies in general., These requirements must ensure that only additional data needed in the project is provided and that Research Services get the information necessary to be able to handle the additional microdata and make it available in the specific project., Use of additional data must comply with the same rules regarding confidentiality and transfer as those that apply to Statistics Denmark’s microdata. , Read more about the rules on transfer and sanctioning, Any other data must be provided safely to Statistics Denmark. , You will find the guide for uploading additional data sets under ‘Use of FSE upload’, Requirements for additional data, Only approved and documented data:, Additional data must be covered by an approved project proposal. Documentation of the additional data content must be uploaded as an appendix in Denmark’s Data Portal. See below under ‘Documentation of additional data’., No information identifying individuals or businesses:, Personal names, company names, responses with free text in surveys and other information roughly identifying individuals or businesses is not allowed to be included in additional data. Such variables must be removed or categorised before additional data is provided to Research Services., Only the required key variables:, You may include only the key variables, e.g. civil registration number, required to link the additional data with the other data on the project. Key variables that are not needed must be removed before data is provided to Research Services., Only numeric variables and categorised text variables:, Additional data may not include non-categorised text variables, such as free text., Only information required for the specific project:, Key variables for which there is no need, must be removed before data is provided to Research Services., File formats, Additional data may be provided in the following formats:, .ASC, .CSV, .DTA, .ODS, .PDF, .SAS7BCAT, .SAS7BDAT, .SAV, .XLS, .XLSX, Documentation of additional data, Additional data must be documented, so that Research Services gets the information that is necessary to be able to handle the additional data and provide it for the specific project. For that reason, the institution is responsible for uploading an overview with the below content as an appendix in Denmark’s Data Portal., A short description of data (for example origin and content), Name of the data set, names of variables and a description of the variable content., The key variables that must be de-identified. This means the variables required to link the additional data with the other data in the project (e.g. civil registration number)., Which additional variables must be de-identified? This means variables that can be attributed directly to individuals or businesses (e.g. CVR number, grant number for health practitioner, serial number, or other ID numbers)., Does the additional data include key variables that must be linked with key variables in previously provided data sets?, If data is provided directly to Research Services from another data provider, further documentation may be necessary, e.g. variables that must be deleted (see ‘Requirements for additional data’ above)., The overview must be uploaded as an appendix in Denmark’s Data Portal under ’Additional data sources’ in a generally available format (Excel, Word or similar)., If data is to be linked with more populations, the documentation must be attached as an appendix under ‘Additional data sources’ for minimum one of these populations. The additional data must appear under ‘Additional data sources’ for each population., When the additional data has been provided to Statistics Denmark and the documentation has been uploaded in Denmark’s Data Portal, it is recommended that you notify the project owner in Research Services via email., Prices for delivery of submitted data, The price for the delivery of submitted data is variable. Once we have received the data, we prepare a framework agreement with a maximum expected time consumption. After delivery of the data, we settle based on the actual time consumption. , See more about framework agreements under Prices and price agreements, ., The expected time consumption depends on many factors, including the number, size, complexity and format of the files. Therefore, we always prepare framework agreements for submitted data on a case-by-case basis, and only when we have received the specific data sets., The table below provides a guide to the typical relationship between the number of data sets and the maximum expected time consumption. Please note, however, that the number of data sets is only one of many factors, and the table is therefore only a guide. Other factors, such as size, number of variables, or special treatment, may increase or decrease the number of hours in the specific framework agreement. Contact the project manager in Research Services if you need a more precise estimate., See the current hourly rates under Prices and price agreements, Number of datasets, Typical number of hours in framework agreement (indicative), 1-2, 2-4, 3-10, 4-8, 11-20, 8-14, 21-50, 14-20, NOTE: The number of hours in the framework agreement is typically set to accommodate unforeseen circumstances. Since the settlement is based on the actual time spent, the final price will in many cases be lower., Submitting files that do not include microdata, Files that you need in your project and that do not include microdata (e.g. programme files), must not be uploaded via FSE Upload., The files can be sent directly to the project owner in Research Services, if you are working on a subproject for a project database. If you are working under the researcher scheme, you can send an email to , forskningsservice@dst.dk, . In both instances, you must attach the files to the email. Further, you must:, Confirm that you have checked the files to ensure that they do not include microdata., Confirm that the files do not include microdata., Provide a short description of the content of the files and its relevance in terms of the purpose of the project., Indicate the specific path to where the files must be located., Ensure that the files are submitted in a generally available format that can be opened and checked by Research Services without use of specialised software., , De-identification, When the additional data has been received in Research Services it will be de-identified in the same way as any other data that belongs to the project. This happens by de-identifying key variables. Subsequently, the additional data is made available together with any other data in the project., Providing additional data to Research Services, Additional data must be provided safely to Statistics Denmark, Additional data can be provided safely to Statistics Denmark in the ways stated below:, Use of FSE Upload, Under Statistics Denmark’s microdata schemes, it is possible to upload additional files with data to be used in an existing project. Only data and documentation can be uploaded. Programmes etc. can be sent to the Research Services employee who is responsible for the project (project owner)., The documentation must either be sent to the project owner from Statistics Denmark or be uploaded in one of the allowed file formats., Before you upload, Before you upload data to be used in a project, you must ensure that the criteria for use of your data in the project have been met. The requirements are described at the top of the page under ‘Requirements for additional data’., The handling of additional data is invoiced according to the actual time used, unless otherwise agreed., Contact the project owner in Statistics Denmark in advance., How to upload the files, You log in via remote.dst.dk in the same way as when you are going to work on a project., Under 'Applications and Links' select 'FSE-UPLOAD'., Write the project number of the project where your data is to be used., When the project title is shown next to the project number, you must check that you have selected the right project for uploading of data., If relevant, add a comment concerning your data in the comments field for the project owner at Statistics Denmark. Actual correspondence should take place via email. , Add files to be uploaded by clicking `Add file'. You can add one or more files. Each file may run up to 2 GB., Note that not all types of files can be uploaded. You can see the list of allowed types under `File formats' further up on the page., When all files have been selected, you must click `Upload'., When all the files have been uploaded, you can log out in the bottom right-hand corner., If relevant, see , this guide for FSE upload (pdf, in Danish), , which includes screenshots of the upload process., Use of secure email, Data with a max. file size of 100 MB can be sent via secure email to , forskerpost@dst.dk, in one of two ways: , The institution retrieves Statistics Denmark's certificate at the website for download of security certificates (in Danish): Select , forskerpost@dst.dk, . For Outlook, we recommend the Vcf format. , After this, additional data must be sent to , forskerpost@dst.dk, . Always indicate project number and project owner in Research Services in the subject field of the email and notify the project owner in Research Services directly, once the additional data has been sent. Statistics Denmark does not offer any support for encryption or digital signature. Refer to your own IT department for guidance using mitID and secure email., The institution can use a secure, encrypted tunnel (SEPO). The set-up is individual for different institutions and must be handled by the institution's IT department, which should be involved before the additional data is sent., You must always indicate the project number and notify the project owner in Research Services that the email has been sent using an encrypted tunnel (SEPO). If using an encrypted tunnel, additional data must also be sent to , forskerpost@dst.dk, ., Registered letter or personal delivery, For data security reasons, we recommend that you use one of the above options., If this is not possible (e.g. if the files are very big), it is possible to provide password-protected additional data on physical media directly to Statistics Denmark at the below address. The contact person for the project in Research Services and the project number must always appear from the material handed in:, Statistics Denmark, Service Desk, Sankt Kjelds Plads 11, 2100 Copenhagen Ø, Att. Contact person for the project in Research Services, Re. project number: 7XXXXX, Additional data can either be delivered personally at Statistics Denmark's reception or be sent by registered mail to Statistics Denmark on a physical medium (DVD, CD-ROM or USB, which will not be returned). When provided on a physical medium, the additional data must be password-protected. Password must not be provided together with the physical medium., When the additional data has been received, you must send the password via email to the contact person for the project in Research Services., , Other data providers, Other data providers can provide additional data directly to Statistics Denmark at the request of the institution and by agreement between Research Services and the data provider. Delivery of additional data must take place in one of the above ways, but you must make sure that the additional data complies with the requirements above before it are delivered to Research Services., Do , not , send data via standard email, Additional data must not be sent via standard email, since this is not a secure delivery mode.

https://www.dst.dk/en/TilSalg/data-til-forskning/anmodning-om-data/tilknytning-af-ovrige-data