Data science: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
Restored revision 1220648548 by Grayfell (talk): Rv recent additions that depend on unreliable sourcing
BSarara (talk | contribs)
9 edits
I added content only from sources(journals) that I researched, added citations and figures from projects I have worked on previously. None of the content except for the pictures (and rephrasing the sources) is my own work.
Tags: Reverted Visual edit
Line 51: Line 51:


In summary, data analysis and data science are distinct yet interconnected disciplines within the broader field of [[data management]] and analysis. Data analysis focuses on extracting insights and drawing conclusions from [[structured data]], while data science involves a more comprehensive approach that combines [[statistical analysis]], computational methods, and [[machine learning]] to extract insights, build predictive models, and drive data-driven [[decision-making]]. Both fields use data to understand patterns, make informed decisions, and solve complex problems across various domains.
In summary, data analysis and data science are distinct yet interconnected disciplines within the broader field of [[data management]] and analysis. Data analysis focuses on extracting insights and drawing conclusions from [[structured data]], while data science involves a more comprehensive approach that combines [[statistical analysis]], computational methods, and [[machine learning]] to extract insights, build predictive models, and drive data-driven [[decision-making]]. Both fields use data to understand patterns, make informed decisions, and solve complex problems across various domains.

== Data Visualization ==
In today's era of [[big data]], where immense quantities of information are continually generated, the capacity to convey insights and identify patterns within this abundance of data has become essential.<ref name=":12">{{Cite journal |last=M |first=Islam |date=2019 |title="An Overview of Data Visualization," 2019 International Conference on Information Science and Communications Technologies (ICISCT), Tashkent, Uzbekistan, 2019, pp. 1-7, doi: 10.1109/ICISCT47635.2019.9012031. |url=https://doi.org/10.1109/ICISCT47635.2019.9012031 |journal=[[An Overview of Data Visualization]]}}</ref> [[Data and information visualization|Data visualization]] plays a crucial role in this process by transforming complex and abstract data into [[Visualization (graphics)|visual representations]] that are intuitive and easy to comprehend.<ref>{{Cite journal |date=June 22, 2016 |title=Weissgerber TL, Garovic VD, Savic M, Winham SJ, Milic NM (2016) From Static to Interactive: Transforming Data Visualization to Improve Transparency. PLoS Biol 14(6): e1002484. |url=https://doi.org/10.1371/journal.pbio.1002484 |journal=[[From Static to Interactive: Transforming Data Visualization to Improve Transparency]]}}</ref>
[[File:Scatter plot of detection frequencies for Nuclear Profiles.png|thumb|The [[scatter plot]] visualizes the detection frequencies of more than 400 Nuclear profiles of the Mouse genome. The x-axis represents different Nuclear profiles, while the y-axis shows their detection frequencies. Each blue dot corresponds to a Nuclear profile, and the red dashed line indicates a threshold or average value. This graph provides valuable insights into the distribution and frequency of Nuclear profiles using Matplotlib.]]
Data Visualization holds a significant position in data science for several reasons, including the following main aspects:

* '''Simplification''': Data visualization transforms complex data sets into a visual context that’s easier for the human mind to grasp. It enables data scientists to discern patterns, trends, and insights that might be overlooked in text-based data.<ref name=":13">{{Cite journal |title=Unwin, A. (2020). Why Is Data Visualization Important? What Is Important in Data Visualization? Harvard Data Science Review, 2(1). |url=https://hdsr.mitpress.mit.edu/pub/zok97i7p/release/4 |journal=[[Why Is Data Visualization Important? What Is Important in Data Visualization? ]]}}</ref>
* '''Helpful [[decision-making]]''': By representing data visually, data visualization allows for faster data interpretation, leading to quicker and more informed decision-making.<ref>{{Cite journal |date=25 August 2021 |title=Eberhard, K. The effects of visualization on judgment and decision-making: a systematic literature review. Manag Rev Q 73, 167–214 (2023). |url=https://doi.org/10.1007/s11301-021-00235-8 |journal=[[The effects of visualization on judgment and decision-making: a systematic literature review]]}}</ref><ref name=":14">{{Cite journal |title=Embarak, O. (2018). The Importance of Data Visualization in Business Intelligence. In: Data Analysis and Visualization Using Python. Apress, Berkeley, CA. |url=https://doi.org/10.1007/978-1-4842-4109-7_2 |journal=[[The Importance of Data Visualization in Business Intelligence. In: Data Analysis and Visualization Using Python]]}}</ref>
* '''Data Visualization in [[Exploratory data analysis|Exploratory Data Analysis (EDA)]]:''' Data visualization is often used in exploratory data analysis, where visualizations can help identify patterns, correlations, [[Outlier|outliers]], or trends in the data that may not be immediately apparent from the raw data.<ref name=":13" /><ref>{{Cite journal |title=Qin, X., Luo, Y., Tang, N. et al. Making data visualization more efficient and effective: a survey. The VLDB Journal 29, 93–117 (2020). |url=https://doi.org/10.1007/s00778-019-00588-3 |journal=[[Making data visualization more efficient and effective: a survey]]}}</ref>

=== '''Data Visualization Tools and Libraries''' ===
Data visualization in Data Science is facilitated by a variety of tools and [[Library (computing)|libraries]] that cater to different needs and preferences. Here are some common ones:

* '''[[Matplotlib]]''': It is a widely used data visualization library in [[Python Package Index|Python]], was developed by John Hunter and several other contributors. Their significant efforts have made this software a go-to tool for [[Scientist|scientists]] and [[philosophers]] worldwide. As an integral part of the Python data science stack, Matplotlib is compatible with [[NumPy]], [[Pandas (software)|Pandas]], and other relevant libraries, making it a comprehensive package for data visualization.<ref name=":14" /><ref>{{Cite journal |title=Hafeez, Abdul & Sial, Ali. (2021). Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python [HEC Y Cat]. International Journal of Advanced Trends in Computer Science and Engineering. |url=https://doi.org/10.30534/ijatcse/2021/391012021 |journal=[[Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python]]}}</ref>

* '''Seaborn''': Seaborn is a data visualization library that builds upon the foundational structures of Matplotlib. It offers users access to common data visualization processes, catering to specific needs such as color mapping to a variable. Seaborn is particularly well-integrated for working with Pandas [[Data Frame|DataFrames]], making it a convenient tool for data visualization tasks.<ref>{{Cite journal |title=Waskom, M. L., (2021). seaborn: statistical data visualization. Journal of Open Source Software, 6(60), 3021 |url=https://doi.org/10.21105/joss.03021 |journal=[[Seaborn: statistical data visualization]]}}</ref><ref>{{Cite journal |title=Hafeez, Abdul & Sial, Ali. (2021). Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python [HEC Y Cat]. International Journal of Advanced Trends in Computer Science and Engineering. 10. 2770-281. 10.30534/ijatcse/2021/391012021. |url=https://doi.org/10.30534/ijatcse/2021/391012021 |journal=[[Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python]]}}</ref>

* '''[[Plotly]]:''' Plotly is a technical computing company that develops online data analytics and visualization tools. Plotly provides online graphing, analytics, and statistics tools and also scientific graphing libraries for Python, [[R (programming language)|R]], [[MATLAB]], [[Perl Compatible Regular Expressions|Perl]], [[Arduino]], and [[RE1-silencing transcription factor|REST1]].<ref name=":14" /><ref>{{Cite journal |date=June 2021 |title=Ran Li, Usama Bilal, Interactive Web-Based Data Visualization with R, Plotly, and Shiny (Carson Sievert), Biometrics |url=https://doi.org/10.1111/biom.13474 |journal=[[Interactive Web-Based Data Visualization with R, Plotly, and Shiny (Carson Sievert)]] |volume=77 |issue=2 |pages=776–777}}</ref>

[[File:Average Percentage of Genomic Windows that contain the features across all NPs in each cluster.png|thumb|The [[radar chart]] showcases a range of genomic features such as Hist1, LAD, CTCF, etc., each represented on a scale from 0 to 60. This scale illustrates the percentage of genomic windows that encompass these features in the Hist1 region. The chart offers a comparative analysis of these features across three distinct clusters. This visualization was created using Plotly.]]

* '''[[Tableau Software|Tableau]]''': Tableau is a leading data visualization tool that transforms raw data into interactive and shareable dashboards. It is designed to make data analysis accessible and intuitive for users across various skill levels, thereby empowering individuals and organizations to derive insights that drive informed decision-making.<ref>{{Cite journal |title=Batt, Steven, Tara Grealis, Oskar Harmon, and Paul Tomolonis. “Learning Tableau: A Data Visualization Tool.” The Journal of Economic Education 51, no. 3–4 (2020): 317–28. |url=https://doi.org/10.1080/00220485.2020.1804503 |journal=[[Learning Tableau: A Data Visualization Tool.]]}}</ref>

In Data Science, various types of data visualizations are used, each serving a unique purpose and ideal for certain use cases<ref name=":13" />. Some general types include [[Graph database|Graph]], [[Chart]], [[Plot (graphics)|plot]], [[Table (database)|Table]], [[Venn diagram|Venn diagrams]]. Creating effective visualizations involves selecting the right type of graph based on the data, using [[Color scheme|color strategically]], [[Labelling|labeling]] all elements clearly, ensuring accurate [[Scaling (geometry)|scaling]] and [[Normalization (statistics)|normalization]], maintaining simplicity for clarity, and always considering the background and the need.<ref name=":12" /><ref>{{Cite journal |last=Schmidt |first=Johanna |date=2020 |title=Schmidt, Johanna. "Usage of Visualization Techniques in Data Science Workflows." VISIGRAPP (3: IVAPP). 2020. |url=https://www.scitepress.org/PublishedPapers/2020/91819/91819.pdf |journal=[[Usage of Visualization Techniques in Data Science Workflows.]]}}</ref>
[[File:Network graph of Genomic window 22800000-22830000.png|thumb|416x416px|The [[Network theory|network graph]] illustrates the interconnections among various genomic windows in a hub (22800000-22830000), determined by the number of connections and the degree centrality for each window. Constructed with Plotly, the size of each node/window is dictated by its degree centrality, while the color is determined by the number of connections. The color gradient scale facilitates easy identification of groups with similar formations.]]
Here is a comprehensive list of wide range of techniques across various fields:

# [[Area chart|Area Charts]]
# [[Bar chart|Bar Charts]]
# [[Box plot|Boxplots]]
# [[Bubble chart|Bubble Charts]]
# [[Bullet graph|Bullet Graphs]]
# [[Candlestick chart|Candlestick Charts]]
# [[Funnel plot|Funnel Plot]]
# [[Gantt chart|Gantt Charts]]
# [[Heat map|Heatmaps]]
# [[Histogram|Histograms]]
# [[Line graph of a hypergraph|Line Graphs]]
# [[Network theory|Network Graphs]]
# [[Parallel coordinates|Parallel Coordinates]]
# [[Pie chart|Pie Charts]]
# [[Radar chart|Radar/Spider Charts/Kiviat Diagrams]]
# [[Scatter plot|Scatter Plots]]
# [[Streamgraph|Stream Graphs]]
# [[Sunburst chart|Sunburst Diagrams]]
# [[Treemapping|Tree Maps]]
# [[Violin plot|Violin Plots]]
# [[Venn diagram|Venn Diagrams]]
# [[:File:Square Pie Chart - Waffle Chart.jpg|Waffle Charts]]
# [[Waterfall chart|Waterfall Charts]]
# [[Tag cloud|Word Clouds]]
# [[Recurrence plot|Pair Plots]]

=== '''Future directions and its challenges''' ===

* Automated visualizations: This typically talks about [[Artificial intelligence|AI]] and [[Machine learning]] to generate visual representations of data. The challenge lies in ensuring that these are transparent, interpretable, and trustworthy.<ref>{{Cite journal |last=Dibia |first=Victor |date=July 2023 |title="{LIDA}: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models |url=https://aclanthology.org/2023.acl-demo.11", |journal=[[Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)]] |volume=3}}</ref>
* [[Big data|Big data analysis]]: This involves processing large volumes of data to uncover hidden patterns, correlations, and other insights. The challenge would be when data processing software is insufficient to handle big data, and visualizing this data in a meaningful way may not be straightforward.<ref>{{Cite journal |last=Wang |first=Lidong |last2=Wang |first2=Guanghui |last3=Alexander |first3=Cheryl Ann |date=2015 |title=Big Data and Visualization: Methods, Challenges and Technology Progress. Digital Technologies. No. 1, 2015, pp 33-38. |url=https://pubs.sciepub.com/dt/1/1/7/ |journal=[[Big Data and Visualization: Methods, Challenges and Technology Progress]] |volume=1}}</ref>
* [[Real-time database|Real-time data visualizations]]: Real-time data visualizations necessitate the display of data as it is being [[Real-time database|updated in real-time]]. The challenge here is to manage the [[Continuous or discrete variable|continuous]] flow of data and update the visualization correspondingly. Nevertheless, the development of more efficient [[Algorithm|algorithms]] and the application of [[Sampling (statistics)|data sampling]] and preserving the integrity of the data could potentially mitigate these challenges.<ref>{{Cite journal |last=P. |first=Chopade |last2=J. |first2=Zhan |last3=K. |first3=Roy |last4=K. |first4=Flurchick |date=2015 |title="Real-time large-scale big data networks analytics and visualization architecture," 2015 12th International Conference & Expo on Emerging Technologies for a Smarter World (CEWIT), Melville, NY, USA, 2015, pp. 1-6, doi: 10.1109/CEWIT.2015.7338157. |url=https://ieeexplore.ieee.org/abstract/document/7338157/citations#citations |journal=[[Real-time large-scale big data networks analytics and visualization architecture]]}}</ref>


== Cloud Computing for Data Science ==
== Cloud Computing for Data Science ==

Revision as of 14:47, 25 April 2024

Not to be confused with information science.

The existence of Comet NEOWISE (here depicted as a series of red dots) was discovered by analyzing astronomical survey data acquired by a space telescope, the Wide-field Infrared Survey Explorer.

Data science is an interdisciplinary academic field[1] that uses statistics, scientific computing, scientific methods, processes, algorithms and systems to extract or extrapolate knowledge and insights from potentially noisy, structured, or unstructured data.[2]

Data science also integrates domain knowledge from the underlying application domain (e.g., natural sciences, information technology, and medicine).[3] Data science is multifaceted and can be described as a science, a research paradigm, a research method, a discipline, a workflow, and a profession.[4]

Data science is "a concept to unify statistics, data analysis, informatics, and their related methods" to "understand and analyze actual phenomena" with data.[5] It uses techniques and theories drawn from many fields within the context of mathematics, statistics, computer science, information science, and domain knowledge.[6] However, data science is different from computer science and information science. Turing Award winner Jim Gray imagined data science as a "fourth paradigm" of science (empirical, theoretical, computational, and now data-driven) and asserted that "everything about science is changing because of the impact of information technology" and the data deluge.[7][8]

A data scientist is a professional who creates programming code and combines it with statistical knowledge to create insights from data.[9]

Foundations

Data science is an interdisciplinary field[10] focused on extracting knowledge from typically large data sets and applying the knowledge and insights from that data to solve problems in a wide range of application domains. The field encompasses preparing data for analysis, formulating data science problems, analyzing data, developing data-driven solutions, and presenting findings to inform high-level decisions in a broad range of application domains. As such, it incorporates skills from computer science, statistics, information science, mathematics, data visualization, information visualization, data sonification, data integration, graphic design, complex systems, communication and business.[11][12] Statistician Nathan Yau, drawing on Ben Fry, also links data science to human–computer interaction: users should be able to intuitively control and explore data.[13][14] In 2015, the American Statistical Association identified database management, statistics and machine learning, and distributed and parallel systems as the three emerging foundational professional communities.[15]

Relationship to statistics

Many statisticians, including Nate Silver, have argued that data science is not a new field, but rather another name for statistics.[16] Others argue that data science is distinct from statistics because it focuses on problems and techniques unique to digital data.[17] Vasant Dhar writes that statistics emphasizes quantitative data and description. In contrast, data science deals with quantitative and qualitative data (e.g., from images, text, sensors, transactions, customer information, etc.) and emphasizes prediction and action.[18] Andrew Gelman of Columbia University has described statistics as a non-essential part of data science.[19]

Stanford professor David Donoho writes that data science is not distinguished from statistics by the size of datasets or use of computing and that many graduate programs misleadingly advertise their analytics and statistics training as the essence of a data-science program. He describes data science as an applied field growing out of traditional statistics.[20]

Etymology

Early usage

In 1962, John Tukey described a field he called "data analysis", which resembles modern data science.[20] In 1985, in a lecture given to the Chinese Academy of Sciences in Beijing, C. F. Jeff Wu used the term "data science" for the first time as an alternative name for statistics.[21] Later, attendees at a 1992 statistics symposium at the University of Montpellier  II acknowledged the emergence of a new discipline focused on data of various origins and forms, combining established concepts and principles of statistics and data analysis with computing.[22][23]

The term "data science" has been traced back to 1974, when Peter Naur proposed it as an alternative name to computer science.[6] In 1996, the International Federation of Classification Societies became the first conference to specifically feature data science as a topic.[6] However, the definition was still in flux. After the 1985 lecture at the Chinese Academy of Sciences in Beijing, in 1997 C. F. Jeff Wu again suggested that statistics should be renamed data science. He reasoned that a new name would help statistics shed inaccurate stereotypes, such as being synonymous with accounting or limited to describing data.[24] In 1998, Hayashi Chikio argued for data science as a new, interdisciplinary concept, with three aspects: data design, collection, and analysis.[23]

During the 1990s, popular terms for the process of finding patterns in datasets (which were increasingly large) included "knowledge discovery" and "data mining".[6][25]

Modern usage

In 2012, technologists Thomas H. Davenport and DJ Patil declared "Data Scientist: The Sexiest Job of the 21st Century",[26] a catchphrase that was picked up even by major-city newspapers like the New York Times[27] and the Boston Globe.[28] A decade later, they reaffirmed it, stating that "the job is more in demand than ever with employers".[29]

The modern conception of data science as an independent discipline is sometimes attributed to William S. Cleveland.[30] In a 2001 paper, he advocated an expansion of statistics beyond theory into technical areas; because this would significantly change the field, it warranted a new name.[25] "Data science" became more widely used in the next few years: in 2002, the Committee on Data for Science and Technology launched the Data Science Journal. In 2003, Columbia University launched The Journal of Data Science.[25] In 2014, the American Statistical Association's Section on Statistical Learning and Data Mining changed its name to the Section on Statistical Learning and Data Science, reflecting the ascendant popularity of data science.[31]

The professional title of "data scientist" has been attributed to DJ Patil and Jeff Hammerbacher in 2008.[32] Though it was used by the National Science Board in their 2005 report "Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century", it referred broadly to any key role in managing a digital data collection.[33]

There is still no consensus on the definition of data science, and it is considered by some to be a buzzword.[34] Big data is a related marketing term.[35] Data scientists are responsible for breaking down big data into usable information and creating software and algorithms that help companies and organizations determine optimal operations.[36]

Data science and data analysis

summary statistics and scatterplots showing the Datasaurus dozen data set
Example for the usefulness of exploratory data analysis as demonstrated using the Datasaurus dozen data set

Data science and data analysis are both important disciplines in the field of data management and analysis, but they differ in several key ways. While both fields involve working with data, data science is more of an interdisciplinary field that involves the application of statistical, computational, and machine learning methods to extract insights from data and make predictions, while data analysis is more focused on the examination and interpretation of data to identify patterns and trends.[37][38]

Data analysis typically involves working with smaller, structured datasets to answer specific questions or solve specific problems. This can involve tasks such as data cleaning, data visualization, and exploratory data analysis to gain insights into the data and develop hypotheses about relationships between variables. Data analysts typically use statistical methods to test these hypotheses and draw conclusions from the data. For example, a data analyst might analyze sales data to identify trends in customer behavior and make recommendations for marketing strategies.[37]

Data science, on the other hand, is a more complex and iterative process that involves working with larger, more complex datasets that often require advanced computational and statistical methods to analyze. Data scientists often work with unstructured data such as text or images and use machine learning algorithms to build predictive models and make data-driven decisions. In addition to statistical analysis, data science often involves tasks such as data preprocessing, feature engineering, and model selection. For instance, a data scientist might develop a recommendation system for an e-commerce platform by analyzing user behavior patterns and using machine learning algorithms to predict user preferences.[38][39]

While data analysis focuses on extracting insights from existing data, data science goes beyond that by incorporating the development and implementation of predictive models to make informed decisions. Data scientists are often responsible for collecting and cleaning data, selecting appropriate analytical techniques, and deploying models in real-world scenarios. They work at the intersection of mathematics, computer science, and domain expertise to solve complex problems and uncover hidden patterns in large datasets.[38]

Despite these differences, data science and data analysis are closely related fields and often require similar skill sets. Both fields require a solid foundation in statistics, programming, and data visualization, as well as the ability to communicate findings effectively to both technical and non-technical audiences. Both fields benefit from critical thinking and domain knowledge, as understanding the context and nuances of the data is essential for accurate analysis and modeling.[37][38]

In summary, data analysis and data science are distinct yet interconnected disciplines within the broader field of data management and analysis. Data analysis focuses on extracting insights and drawing conclusions from structured data, while data science involves a more comprehensive approach that combines statistical analysis, computational methods, and machine learning to extract insights, build predictive models, and drive data-driven decision-making. Both fields use data to understand patterns, make informed decisions, and solve complex problems across various domains.

Data Visualization

In today's era of big data, where immense quantities of information are continually generated, the capacity to convey insights and identify patterns within this abundance of data has become essential.[40] Data visualization plays a crucial role in this process by transforming complex and abstract data into visual representations that are intuitive and easy to comprehend.[41]

The scatter plot visualizes the detection frequencies of more than 400 Nuclear profiles of the Mouse genome. The x-axis represents different Nuclear profiles, while the y-axis shows their detection frequencies. Each blue dot corresponds to a Nuclear profile, and the red dashed line indicates a threshold or average value. This graph provides valuable insights into the distribution and frequency of Nuclear profiles using Matplotlib.

Data Visualization holds a significant position in data science for several reasons, including the following main aspects:

  • Simplification: Data visualization transforms complex data sets into a visual context that’s easier for the human mind to grasp. It enables data scientists to discern patterns, trends, and insights that might be overlooked in text-based data.[42]
  • Helpful decision-making: By representing data visually, data visualization allows for faster data interpretation, leading to quicker and more informed decision-making.[43][44]
  • Data Visualization in Exploratory Data Analysis (EDA): Data visualization is often used in exploratory data analysis, where visualizations can help identify patterns, correlations, outliers, or trends in the data that may not be immediately apparent from the raw data.[42][45]

Data Visualization Tools and Libraries

Data visualization in Data Science is facilitated by a variety of tools and libraries that cater to different needs and preferences. Here are some common ones:

  • Matplotlib: It is a widely used data visualization library in Python, was developed by John Hunter and several other contributors. Their significant efforts have made this software a go-to tool for scientists and philosophers worldwide. As an integral part of the Python data science stack, Matplotlib is compatible with NumPy, Pandas, and other relevant libraries, making it a comprehensive package for data visualization.[44][46]
  • Seaborn: Seaborn is a data visualization library that builds upon the foundational structures of Matplotlib. It offers users access to common data visualization processes, catering to specific needs such as color mapping to a variable. Seaborn is particularly well-integrated for working with Pandas DataFrames, making it a convenient tool for data visualization tasks.[47][48]
  • Plotly: Plotly is a technical computing company that develops online data analytics and visualization tools. Plotly provides online graphing, analytics, and statistics tools and also scientific graphing libraries for Python, R, MATLAB, Perl, Arduino, and REST1.[44][49]
The radar chart showcases a range of genomic features such as Hist1, LAD, CTCF, etc., each represented on a scale from 0 to 60. This scale illustrates the percentage of genomic windows that encompass these features in the Hist1 region. The chart offers a comparative analysis of these features across three distinct clusters. This visualization was created using Plotly.
  • Tableau: Tableau is a leading data visualization tool that transforms raw data into interactive and shareable dashboards. It is designed to make data analysis accessible and intuitive for users across various skill levels, thereby empowering individuals and organizations to derive insights that drive informed decision-making.[50]

In Data Science, various types of data visualizations are used, each serving a unique purpose and ideal for certain use cases[42]. Some general types include Graph, Chart, plot, Table, Venn diagrams. Creating effective visualizations involves selecting the right type of graph based on the data, using color strategically, labeling all elements clearly, ensuring accurate scaling and normalization, maintaining simplicity for clarity, and always considering the background and the need.[40][51]

The network graph illustrates the interconnections among various genomic windows in a hub (22800000-22830000), determined by the number of connections and the degree centrality for each window. Constructed with Plotly, the size of each node/window is dictated by its degree centrality, while the color is determined by the number of connections. The color gradient scale facilitates easy identification of groups with similar formations.

Here is a comprehensive list of wide range of techniques across various fields:

  1. Area Charts
  2. Bar Charts
  3. Boxplots
  4. Bubble Charts
  5. Bullet Graphs
  6. Candlestick Charts
  7. Funnel Plot
  8. Gantt Charts
  9. Heatmaps
  10. Histograms
  11. Line Graphs
  12. Network Graphs
  13. Parallel Coordinates
  14. Pie Charts
  15. Radar/Spider Charts/Kiviat Diagrams
  16. Scatter Plots
  17. Stream Graphs
  18. Sunburst Diagrams
  19. Tree Maps
  20. Violin Plots
  21. Venn Diagrams
  22. Waffle Charts
  23. Waterfall Charts
  24. Word Clouds
  25. Pair Plots

Future directions and its challenges

  • Automated visualizations: This typically talks about AI and Machine learning to generate visual representations of data. The challenge lies in ensuring that these are transparent, interpretable, and trustworthy.[52]
  • Big data analysis: This involves processing large volumes of data to uncover hidden patterns, correlations, and other insights. The challenge would be when data processing software is insufficient to handle big data, and visualizing this data in a meaningful way may not be straightforward.[53]
  • Real-time data visualizations: Real-time data visualizations necessitate the display of data as it is being updated in real-time. The challenge here is to manage the continuous flow of data and update the visualization correspondingly. Nevertheless, the development of more efficient algorithms and the application of data sampling and preserving the integrity of the data could potentially mitigate these challenges.[54]

Cloud Computing for Data Science

A cloud-based architecture for enabling big data analytics. Data flows from various sources, such as personal computers, laptops, and smart phones, through cloud services for processing and analysis, finally leading to various big data applications.

Cloud computing can offer access to large amounts of computational power and storage.[55] In big data, where volumes of information are continually generated and processed, these platforms can be used to handle complex and resource-intensive analytical tasks.[56]

Some distributed computing frameworks are designed to handle big data workloads. These frameworks can enable data scientists to process and analyze large datasets in parallel, which can reducing processing times.[57]

Ethical consideration in Data Science

Data science involve collecting, processing, and analyzing data which often including personal and sensitive information. Ethical concerns include potential privacy violations, bias perpetuation, and negative societal impacts [58][59]

Machine learning models can amplify existing biases present in training data, leading to discriminatory or unfair outcomes.[60][61]

See also

References

  1. ^ Donoho, David (2017). "50 Years of Data Science". Journal of Computational and Graphical Statistics. 26 (4): 745–766. doi:10.1080/10618600.2017.1384734. S2CID 114558008.
  2. ^ Dhar, V. (2013). "Data science and prediction". Communications of the ACM. 56 (12): 64–73. doi:10.1145/2500499. S2CID 6107147. Archived from the original on 9 November 2014. Retrieved 2 September 2015.
  3. ^ Danyluk, A.; Leidig, P. (2021). Computing Competencies for Undergraduate Data Science Curricula (PDF). ACM Data Science Task Force Final Report (Report).
  4. ^ Mike, Koby; Hazzan, Orit (20 January 2023). "What is Data Science?". Communications of the ACM. 66 (2): 12–13. doi:10.1145/3575663. ISSN 0001-0782.
  5. ^ Hayashi, Chikio (1 January 1998). "What is Data Science ? Fundamental Concepts and a Heuristic Example". In Hayashi, Chikio; Yajima, Keiji; Bock, Hans-Hermann; Ohsumi, Noboru; Tanaka, Yutaka; Baba, Yasumasa (eds.). Data Science, Classification, and Related Methods. Studies in Classification, Data Analysis, and Knowledge Organization. Springer Japan. pp. 40–51. doi:10.1007/978-4-431-65950-1_3. ISBN 9784431702085.
  6. ^ a b c d Cao, Longbing (29 June 2017). "Data Science: A Comprehensive Overview". ACM Computing Surveys. 50 (3): 43:1–43:42. arXiv:2007.03606. doi:10.1145/3076253. ISSN 0360-0300. S2CID 207595944.
  7. ^ Tony Hey; Stewart Tansley; Kristin Michele Tolle (2009). The Fourth Paradigm: Data-intensive Scientific Discovery. Microsoft Research. ISBN 978-0-9825442-0-4. Archived from the original on 20 March 2017.
  8. ^ Bell, G.; Hey, T.; Szalay, A. (2009). "Computer Science: Beyond the Data Deluge". Science. 323 (5919): 1297–1298. doi:10.1126/science.1170411. ISSN 0036-8075. PMID 19265007. S2CID 9743327.
  9. ^ Davenport, Thomas H.; Patil, D. J. (October 2012). "Data Scientist: The Sexiest Job of the 21st Century". Harvard Business Review. 90 (10): 70–76, 128. PMID 23074866. Retrieved 18 January 2016.
  10. ^ Emmert-Streib, Frank; Dehmer, Matthias (2018). "Defining data science by a data-driven quantification of the community". Machine Learning and Knowledge Extraction. 1: 235–251. doi:10.3390/make1010015.
  11. ^ "1. Introduction: What Is Data Science?". Doing Data Science [Book]. O’Reilly. Retrieved 3 April 2020.
  12. ^ "the three sexy skills of data geeks". m.e.driscoll: data utopian. 27 May 2009. Retrieved 3 April 2020.
  13. ^ Yau, Nathan (4 June 2009). "Rise of the Data Scientist". FlowingData. Retrieved 3 April 2020.
  14. ^ "Basic Example". benfry.com. Retrieved 3 April 2020.
  15. ^ "ASA Statement on the Role of Statistics in Data Science". AmStatNews. American Statistical Association. 1 October 2015. Archived from the original on 20 June 2019. Retrieved 29 May 2019.
  16. ^ "Nate Silver: What I need from statisticians". Statistics Views. Retrieved 3 April 2020.
  17. ^ "What's the Difference Between Data Science and Statistics?". Priceonomics. 13 October 2015. Retrieved 3 April 2020.
  18. ^ Vasant Dhar (1 December 2013). "Data science and prediction". Communications of the ACM. 56 (12): 64–73. doi:10.1145/2500499. S2CID 6107147.
  19. ^ "Statistics is the least important part of data science « Statistical Modeling, Causal Inference, and Social Science". statmodeling.stat.columbia.edu. Retrieved 3 April 2020.
  20. ^ a b Donoho, David (18 September 2015). "50 years of Data Science" (PDF). Retrieved 2 April 2020.
  21. ^ Wu, C. F. Jeff (1986). "Future directions of statistical research in China: a historical perspective" (PDF). Application of Statistics and Management. 1: 1–7. Retrieved 29 November 2020.
  22. ^ Escoufier, Yves; Hayashi, Chikio; Fichet, Bernard, eds. (1995). Data science and its applications. Tokyo: Academic Press/Harcourt Brace. ISBN 0-12-241770-4. OCLC 489990740.
  23. ^ a b Murtagh, Fionn; Devlin, Keith (2018). "The Development of Data Science: Implications for Education, Employment, Research, and the Data Revolution for Sustainable Development". Big Data and Cognitive Computing. 2 (2): 14. doi:10.3390/bdcc2020014.
  24. ^ Wu, C. F. Jeff. "Statistics=Data Science?" (PDF). Retrieved 2 April 2020.
  25. ^ a b c Press, Gil. "A Very Short History of Data Science". Forbes. Retrieved 3 April 2020.
  26. ^ Davenport, Thomas (1 October 2012). "Data Scientist: The Sexiest Job of the 21st Century". Harvard Business Review. Retrieved 10 October 2022.
  27. ^ Miller, Claire (4 April 2013). "Data Science: The Numbers of Our Lives". New York Times. New York City. Retrieved 10 October 2022.
  28. ^ Borchers, Callum (11 November 2015). "Behind the scenes of the 'sexiest job of the 21st century'". Boston Globe. Boston. Retrieved 10 October 2022.
  29. ^ Davenport, Thomas (15 July 2022). "Is Data Scientist Still the Sexiest Job of the 21st Century?". Harvard Business Review. Retrieved 10 October 2022.
  30. ^ Gupta, Shanti (11 December 2015). "William S. Cleveland". Retrieved 2 April 2020.
  31. ^ Talley, Jill (1 June 2016). "ASA Expands Scope, Outreach to Foster Growth, Collaboration in Data Science". Amstat News. American Statistical Association.. In 2013 the first European Conference on Data Analysis (ECDA2013) started in Luxembourg the process which founded the European Association for Data Science (EuADS) www.euads.org in Luxembourg in 2015.
  32. ^ Davenport, Thomas H.; Patil, D. J. (1 October 2012). "Data Scientist: The Sexiest Job of the 21st Century". Harvard Business Review. No. October 2012. ISSN 0017-8012. Retrieved 3 April 2020.
  33. ^ "US NSF – NSB-05-40, Long-Lived Digital Data Collections Enabling Research and Education in the 21st Century". www.nsf.gov. Retrieved 3 April 2020.
  34. ^ Press, Gil. "Data Science: What's The Half-Life of a Buzzword?". Forbes. Retrieved 3 April 2020.
  35. ^ Pham, Peter. "The Impacts of Big Data That You May Not Have Heard Of". Forbes. Retrieved 3 April 2020.
  36. ^ Martin, Sophia (20 September 2019). "How Data Science will Impact Future of Businesses?" (PDF). Medium. Retrieved 3 April 2020.
  37. ^ a b c Gareth, Hastie; Witten, Tibshira (29 September 2017). "An Introduction to Statistical Learning: with Applications in R." Springer.
  38. ^ a b c d Provost, Foster; Tom Fawcett (1 August 2013). "Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking". O'Reilly Media, Inc.
  39. ^ Han, Kamber; Pei (2011). Data Mining: Concepts and Techniques. ISBN 9780123814791.
  40. ^ a b M, Islam (2019). ""An Overview of Data Visualization," 2019 International Conference on Information Science and Communications Technologies (ICISCT), Tashkent, Uzbekistan, 2019, pp. 1-7, doi: 10.1109/ICISCT47635.2019.9012031". An Overview of Data Visualization.
  41. ^ "Weissgerber TL, Garovic VD, Savic M, Winham SJ, Milic NM (2016) From Static to Interactive: Transforming Data Visualization to Improve Transparency. PLoS Biol 14(6): e1002484". From Static to Interactive: Transforming Data Visualization to Improve Transparency. 22 June 2016.
  42. ^ a b c "Unwin, A. (2020). Why Is Data Visualization Important? What Is Important in Data Visualization? Harvard Data Science Review, 2(1)". Why Is Data Visualization Important? What Is Important in Data Visualization? .
  43. ^ "Eberhard, K. The effects of visualization on judgment and decision-making: a systematic literature review. Manag Rev Q 73, 167–214 (2023)". The effects of visualization on judgment and decision-making: a systematic literature review. 25 August 2021.
  44. ^ a b c "Embarak, O. (2018). The Importance of Data Visualization in Business Intelligence. In: Data Analysis and Visualization Using Python. Apress, Berkeley, CA". The Importance of Data Visualization in Business Intelligence. In: Data Analysis and Visualization Using Python.
  45. ^ "Qin, X., Luo, Y., Tang, N. et al. Making data visualization more efficient and effective: a survey. The VLDB Journal 29, 93–117 (2020)". Making data visualization more efficient and effective: a survey.
  46. ^ "Hafeez, Abdul & Sial, Ali. (2021). Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python [HEC Y Cat]. International Journal of Advanced Trends in Computer Science and Engineering". Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python.
  47. ^ "Waskom, M. L., (2021). seaborn: statistical data visualization. Journal of Open Source Software, 6(60), 3021". Seaborn: statistical data visualization.
  48. ^ "Hafeez, Abdul & Sial, Ali. (2021). Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python [HEC Y Cat]. International Journal of Advanced Trends in Computer Science and Engineering. 10. 2770-281. 10.30534/ijatcse/2021/391012021". Comparative Analysis of Data Visualization Libraries Matplotlib and Seaborn in Python.
  49. ^ "Ran Li, Usama Bilal, Interactive Web-Based Data Visualization with R, Plotly, and Shiny (Carson Sievert), Biometrics". Interactive Web-Based Data Visualization with R, Plotly, and Shiny (Carson Sievert). 77 (2): 776–777. June 2021.
  50. ^ "Batt, Steven, Tara Grealis, Oskar Harmon, and Paul Tomolonis. "Learning Tableau: A Data Visualization Tool." The Journal of Economic Education 51, no. 3–4 (2020): 317–28". Learning Tableau: A Data Visualization Tool.
  51. ^ Schmidt, Johanna (2020). "Schmidt, Johanna. "Usage of Visualization Techniques in Data Science Workflows." VISIGRAPP (3: IVAPP). 2020" (PDF). Usage of Visualization Techniques in Data Science Workflows.
  52. ^ Dibia, Victor (July 2023). ", ""{LIDA}: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models". Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 3.
  53. ^ Wang, Lidong; Wang, Guanghui; Alexander, Cheryl Ann (2015). "Big Data and Visualization: Methods, Challenges and Technology Progress. Digital Technologies. No. 1, 2015, pp 33-38". Big Data and Visualization: Methods, Challenges and Technology Progress. 1.
  54. ^ P., Chopade; J., Zhan; K., Roy; K., Flurchick (2015). ""Real-time large-scale big data networks analytics and visualization architecture," 2015 12th International Conference & Expo on Emerging Technologies for a Smarter World (CEWIT), Melville, NY, USA, 2015, pp. 1-6, doi: 10.1109/CEWIT.2015.7338157". Real-time large-scale big data networks analytics and visualization architecture.
  55. ^ Hashem, Ibrahim Abaker Targio; Yaqoob, Ibrar; Anuar, Nor Badrul; Mokhtar, Salimah; Gani, Abdullah; Ullah Khan, Samee. "The rise of "big data" on cloud computing: Review and open research issues". Information Systems. 47: 98–115. doi:10.1016/j.is.2014.07.006.
  56. ^ Qiu, Junfei; Wu, Qihui; Ding, Guoru; Xu, Yuhua; Feng, Shuo. "A survey of machine learning for big data processing". EURASIP Journal on Advances in Signal Processing. 2016 (1). doi:10.1186/s13634-016-0355-x. ISSN 1687-6180.{{cite journal}}: CS1 maint: unflagged free DOI (link)
  57. ^ Armbrust, Michael; Xin, Reynold S.; Lian, Cheng; Huai, Yin; Liu, Davies; Bradley, Joseph K.; Meng, Xiangrui; Kaftan, Tomer; Franklin, Michael J.; Ghodsi, Ali; Zaharia, Matei (27 May 2015). "Spark SQL: Relational Data Processing in Spark". Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM: 1383–1394. doi:10.1145/2723372.2742797. ISBN 978-1-4503-2758-9.
  58. ^ Floridi, Luciano; Taddeo, Mariarosaria (28 December 2016). "What is data ethics?". Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 374 (2083): 20160360. doi:10.1098/rsta.2016.0360. ISSN 1364-503X. PMC 5124072. PMID 28336805.
  59. ^ Mittelstadt, Brent Daniel; Floridi, Luciano. "The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts". Science and Engineering Ethics. 22 (2): 303–341. doi:10.1007/s11948-015-9652-2. ISSN 1353-3452.
  60. ^ Barocas, Solon; Selbst, Andrew D (2016). "Big Data's Disparate Impact". California Law Review. doi:10.15779/Z38BG31.{{cite journal}}: CS1 maint: multiple names: authors list (link)
  61. ^ Caliskan, Aylin; Bryson, Joanna J.; Narayanan, Arvind (14 April 2017). "Semantics derived automatically from language corpora contain human-like biases". Science. 356 (6334): 183–186. doi:10.1126/science.aal4230. ISSN 0036-8075.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Data_science&oldid=1220723429"